VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2026-05-18 09:16:35 +03:00

Author	SHA1	Message	Date
Zakhar Bessarab	1272a7f743	app/vminsert/netstorage: refactor snb rebuild Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 17:23:22 +04:00
Zakhar Bessarab	2b39ee785c	app/vmselect: send static empty node ID for multi-level setup Multi-level vmselect setup is not intended to use storage node IDs, so it is safe to return 0 here. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:23 +04:00
Zakhar Bessarab	842bf78cb1	app/vminsert/netstorage: sync comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:23 +04:00
Zakhar Bessarab	5420989018	app/vminsert/netstorage: reinitialize snb on vmstorage connection restore It is needed to rebuild snb in order to ensure that list of storage nodes and consistent hash are in sync. Updating just consistent hash ring is not safe because it can cause misalignment of indexes of alive nodes in snb.sns and hash slots. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:23 +04:00
Zakhar Bessarab	9ff8b312bb	app/vminsert/netstorage: use correct snb reference Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	130b9cd04e	app/vminsert/netstorage: make linter happy Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	88bfad9535	app/vminsert/netstorage: exclude unavailable nodes from consistent hash on start Exclude unhealthy storage nodes from consistent hash in case persistent storage node IDs are enabled. This is needed in order to avoid uneven distribution of load due to default(uint64(0)) IDs assigned to storage nodes. Remove generating fallback ID from node IP address as this will cause a re-distribution of series once storage node will become available and will change its ID. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	e44c6f38c2	app/vminsert/netstorage: print storage node IDs in logs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	96a62a275a	lib/handshake: use a json payload for metadata exchange Update the handshake to use an arbitrary JSON payload to transfer metadata. Handshake sends the metadata length first as an uint64 and then the metadata itself. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	20b9c8007b	lib/storage: print node ID in startup log Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	3df456dd35	lib/storage: don't save persistent node ID on shutdown It is supposed to be saved right after it was generated, there is no reason re-write it again. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	7402ee0801	lib/storage: fallback to address-based ID Generate an ID based on storage node address if storage node is not available. This is needed in order to prevent uneven load distribution if some storage nodes are not available when vminsert is starting. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	5ac1e77520	lib/storage: save storage ID after init This helps to avoid re-creating a storage ID in case of unclean shutdown. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	71e729f3f8	app/vminsert: disable usage of persistent storage node ID by default This is needed in order to avoid complete data re-sharding after the upgrade to a new version. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:22 +04:00
Zakhar Bessarab	8729ec174b	docs/changelog: add info about persisting vmstorage node ID Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:31:18 +04:00
Zakhar Bessarab	84184b707a	app/cluster: communicate node IDs when performing a handshake Send a node ID of vmstorage as a part of vmselect and vminsert handshakes. Use vmstorage node ID as an identifier for consistent hashing at vminsert. Cluster native endpoints calculate vminsert and vmselect node IDs as a hash of all underlying storage node IDs, so that it will also remain consistent in case of address changes. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:29:20 +04:00
Zakhar Bessarab	41e217423f	lib/storage: store node ID in metadata so that it is included in the backups Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:29:20 +04:00
Zakhar Bessarab	8d8073a24d	lib/storage: add storage node id Generate random node ID on start if it is missing or load from disk. Save to storage on storage shutdown. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-29 12:29:19 +04:00
Aliaksandr Valialkin	00b1ee6b5c	vendor: run `make vendor-update`	2024-07-27 13:52:55 +02:00
Aliaksandr Valialkin	9a3f44e79c	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update` after `efd70b2c52`	2024-07-27 13:51:02 +02:00
Aliaksandr Valialkin	5dc7ec058f	app/vmauth: verify how backend response headers are propagated to vmauth client	2024-07-27 13:45:07 +02:00
Aliaksandr Valialkin	d2a825279b	Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of `MustAddRows` and improve readability (#6629 )" This reverts commit `e280d90e9a`. Reason for revert: the updated code doesn't improve the performance of table.MustAddRows for the typical case when rows contain timestamps belonging to ptws[0]. The performance may be improved in theory for the case when all the rows belong to partiton other than ptws[0], but this partition is automatically moved to ptws[0] by the code at lines `6aad1d43e9/lib/storage/table.go (L287-L298)` , so the next time the typical case will work. Also the updated code makes the code harder to follow, since it introduces an additional level of indirection with non-trivial semantics inside table.MustAddRows - the partition.TimeRangeInPartition() function. This function needs to be inspected and understood when reading the code at table.MustAddRows(). This function depends on minTsInRows and maxTsInRows vars, which are defined and initialized many lines above the partition.TimeRangeInPartition() call. This complicates reading and understanding the code even more. The previous code was using clearer loop over rows with the clear call to partition.HasTimestamp() for every timestamp in the row. The partition.HasTimestamp() call is used in the table.MustAddRows() function multiple times. This makes the use of partition.HasTimestamp() call more consistent, easier to understand and easier to maintain comparing to the mix of partition.HasTimestamp() and partition.TimeRangeInPartition() calls. Aslo, there is no need in documenting some hardcore software engineering refactoring at docs/CHANGLELOG.md, since the docs/CHANGELOG.md is intended for VictoriaMetrics users, who may not know software engineering. The docs/CHANGELOG.md must document user-visible changes, and the docs must be concise and clear for VictoriaMetrics users. See https://docs.victoriametrics.com/contributing/#pull-request-checklist for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6629	2024-07-25 14:43:00 +02:00
Aliaksandr Valialkin	75411db0c7	docs/anomaly-detection/components/models.md: remove `{width="800px"}` in the same was as it was done in `4735d8803e`	2024-07-25 14:42:27 +02:00
Aliaksandr Valialkin	eb2a33b6b3	docs/anomaly-detection: return back absolute links instead of relative links after `ce4cc4cbb2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6677 Relative links in docs are much harder to maintain in consistent state comparing to absolute links: - It is non-trivial to figure out the proper relative link path when creating and editing docs. - Relative links break after moving the doc files to another paths, and it is non-trivial to figure which links are broken after that. See also `f357ee57ef` , `d5809f8e12` and `8cb1822b94`	2024-07-25 14:42:09 +02:00
Aliaksandr Valialkin	f320efd5ad	docs/Makefile: simplify and document `make docs-debug` after `ce4cc4cbb2` - Document `make docs-debug` command at https://docs.victoriametrics.com/#documentation - Remove unneeded ROOTDIR, REPODIR and WORKDIR env vars from docs/Makefile , since it is documented and expected that all the Makefile commands are run from the repository root. - Use `docker --rm` for running Docker container with local docs server, so it is automatically removed after pressing `Ctrl+C`. This makes the container cleanup automatic. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6677	2024-07-25 14:41:41 +02:00
Aliaksandr Valialkin	a135a4dcfa	Revert "removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691 )" This reverts commit `2e9b1efeb9`. Reason for revert: relative links in docs are much harder to maintain in consistent state comparing to absolute links: - It is non-trivial to figure out the proper relative link path when creating and editing docs. - Relative links break after moving the doc files to another paths, and it is non-trivial to figure which links are broken after that. - The updated relative links do not work properly right now in the docs. For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page. This is documented at https://docs.victoriametrics.com/#images-in-documentation .	2024-07-25 14:41:13 +02:00
Aliaksandr Valialkin	83d53dfb0d	Revert "replaced global http refs with relative markdown ones (#6692 )" This reverts commit `537266363a`. Reason for revert: relative links in docs are much harder to maintain in consistent state comparing to absolute links: - It is non-trivial to figure out the proper relative link path when creating and editing docs. - Relative links break after moving the doc files to another paths, and it is non-trivial to figure which links are broken after that. - The updated relative links do not work properly right now in the docs. For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page. This is documented at https://docs.victoriametrics.com/#images-in-documentation .	2024-07-25 14:40:53 +02:00
Aliaksandr Valialkin	edc16e171e	Revert "fixed victorialogs relative links (#6693 )" This reverts commit `683a69d53d`. Reason for revert: relative links in docs are much harder to maintain in consistent state comparing to absolute links: - It is non-trivial to figure out the proper relative link path when creating and editing docs. - Relative links break after moving the doc files to another paths, and it is non-trivial to figure which links are broken after that. - The updated relative links do not work properly right now in the docs. For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page. This is documented at https://docs.victoriametrics.com/#images-in-documentation .	2024-07-25 14:40:31 +02:00
Github Actions	e4195f1b47	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@d25ea09 (#6698 )	2024-07-25 14:39:46 +02:00
Github Actions	d7ce94c20d	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@d6bfe49 (#6697 )	2024-07-25 14:39:16 +02:00
Aliaksandr Valialkin	f17265b67d	docs/vmauth.md: typo fix	2024-07-25 14:38:28 +02:00
Artem Navoiev	1951555ffe	docs: fix link in logsql doc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 14:38:11 +02:00
Ruixiang Tan	8e2ff15203	refactor(vmstorage): Refactor the code to reduce the time complexity of `MustAddRows` and improve readability (#6629 ) ### Describe Your Changes The original logic is not only highly complex but also poorly readable, so it can be modified to increase readability and reduce time complexity. --------- Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>	2024-07-25 13:52:54 +02:00
Andrii Chubatiuk	d25f88b912	fixed victorialogs relative links (#6693 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-25 13:41:19 +02:00
Artem Navoiev	3f85306738	docs: guides remove unneeded menu item Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 13:40:47 +02:00
Artem Navoiev	571ecbb610	docs: anomaly detection remove html tags Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 13:40:25 +02:00
Artem Navoiev	61f9e3112d	docs: remove img width Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 13:38:25 +02:00
Artem Navoiev	85d624d6c2	docs: vmanomaly use absolute links in code comments Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 13:27:40 +02:00
Andrii Chubatiuk	b8d80ddae5	replaced global http refs with relative markdown ones (#6692 ) ### Describe Your Changes Replaced global http links in docs with relative markdown ones ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-25 13:26:51 +02:00
Andrii Chubatiuk	9a051fc80f	removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691 ) ### Describe Your Changes Use relative markdown references, removed `{{< ref >}}` shortcodes ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-25 13:20:05 +02:00
Andrii Chubatiuk	6b97044d8a	view documentation locally (#6677 ) - moved files from root to VictoriaMetrics folder to be able to mount operator docs and VictoriaMetrics docs independently - added ability to run website locally The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-25 12:27:05 +02:00
Andrii Chubatiuk	4a01709b90	added _index.md for hugo, which point to README.md (#6686 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-25 12:20:40 +02:00
Hui Wang	e0c62e5c50	security: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2 (#6684 ) See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html >including security fix for: OpenSSL CVE-2024-5535	2024-07-25 11:02:23 +02:00
Zakhar Bessarab	9f5eb25150	app/vmauth: change response code when all backend are not available (#6676 ) ### Describe Your Changes Change response code to 502 to align it with behaviour of other existing reverse proxies. Currently, the following reverse proxies will return 502 in case an upstream is not available: nginx, traefik, caddy, apache. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-25 10:24:05 +02:00
Zhu Jiekun	bafdc9dd78	docs: [vmagent] Add docs for remote write protocol control in Kafka integration (#6682 ) ### Describe Your Changes Improve documentation to help: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6670 Currently, the documentation of Kafka Integration did not describe: - How to switch between different remote write protocols in producer side and consumer side.	2024-07-25 10:21:37 +02:00
Zakhar Bessarab	53c08cb8d7	docs/vmbackup: add a note for azure identity (#6675 ) ### Describe Your Changes Add a note to clarify usage of azure credentials when multiple credentials are available. The user is required to specify AZURE_CLIENT_ID as otherwise Azure API will return an error: "Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request" ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-25 10:20:50 +02:00
Artem Navoiev	591c42fdbe	docs: fix broken links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-25 10:19:59 +02:00
Aliaksandr Valialkin	b65f32350e	docs/CHANGELOG.md: document `keep_original_host` option at vmauth This option has been added in the commit `add2db12b2`	2024-07-20 11:48:58 +02:00
Aliaksandr Valialkin	339821f5ce	app/vmauth: test how User-Agent header is set in requests to backend	2024-07-20 11:43:36 +02:00
Aliaksandr Valialkin	b31bd5613f	app/vmauth: verify the correctness of X-Forwarded-For header processing at TestRequestHandler()	2024-07-20 11:43:36 +02:00
Aliaksandr Valialkin	8f3fd62f50	app/vmauth: add missing tests for requestHandler()	2024-07-20 11:22:54 +02:00
Aliaksandr Valialkin	37fdba6897	app/vmauth: add more tests for requestHandler()	2024-07-20 10:19:57 +02:00
Aliaksandr Valialkin	28af963940	docs/vmauth.md: document the case with default url_prefix additionally to url_map	2024-07-20 09:46:31 +02:00
Aliaksandr Valialkin	a50a29500f	app/vmauth: properly proxy requests to backend paths ending with / Previously the traling / was incorrectly removed when proxying requests from http://vmauth/ While at it, add more tests for requestHandler()	2024-07-19 17:29:17 +02:00
Aliaksandr Valialkin	4e3acfbe9a	app/vmauth: properly proxy HTTP requests without body The Request.Body for requests without body can be nil. This could break readTrackingBody.Read() logic, which could incorrectly return "cannot read data after closing the reader" error in this case. Fix this by initializing the readTrackingBody.r with zeroReader. While at it, properly set Host header if it is specified in 'headers' section. It must be set net/http.Request.Host instead of net/http.Request.Header.Set(), since the net/http.Client overwrites the Host header with the value from req.Host before sending the request. While at it, add tests for requestHandler(). Additional tests for various requestHandler() cases will be added in future commits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525	2024-07-19 16:26:07 +02:00
Aliaksandr Valialkin	3dd239fb87	docs/goals.md: clarify that complicated CI/CD tasks make debugging harder	2024-07-19 08:51:23 +02:00
Aliaksandr Valialkin	5436911abb	docs/vmauth.md: use /some_path/.* instead of /some_path/.+ in examples, so it matches `/some_path/` The /some_path/.+ regexp matches /some_path/ followed by at least a single char. This is unexpected by most users, since they expect it should match /some_path/. Substitute .+ with .*, so this regexp matches /some_path/ .	2024-07-19 08:51:22 +02:00
Yury Molodov	be2a61c244	vmui/logs: switched requests to sequential execution (#6624 ) ### Describe Your Changes This PR changes `/select/logsql/query` and `/select/logsql/hits` to execute sequentially Fixed https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558#issuecomment-2219298984 ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-18 11:56:19 +02:00
Aliaksandr Valialkin	7eb24d98e1	docs/Single-server-VictoriaMetrics.md: recommend periodic running of `make spellcheck` command This is a follow-up for `fabf0b928e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6562	2024-07-18 11:52:34 +02:00
Aliaksandr Valialkin	12393c64f7	docs/goals.md: make clear that complication of CI/CD pipeline is bad CI/CD pipeline is usually hard to debug. So it is good to keep it as simple as possible.	2024-07-18 11:52:24 +02:00
Andrii Chubatiuk	13b9a2cfc1	Code tooltip updates (#6657 ) ### Describe Your Changes recently added custom `promtextmetric` and `influxtextmetric` prismjs plugins to vmdocs to convert this <img width="1081" alt="Screenshot 2024-07-17 at 12 13 29" src="https://github.com/user-attachments/assets/66d4ea18-10fe-45ef-80b4-989d0eb3bd92"> to this <img width="1087" alt="Screenshot 2024-07-17 at 12 24 34" src="https://github.com/user-attachments/assets/60d4ab44-79e5-4c63-b966-54b989ead1aa"> ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-18 11:52:24 +02:00
Aliaksandr Valialkin	3cda945138	docs/Release-Guide.md: shorten release guide for Ansible roles and RPM packages There is no need in enumerating all the files, which must be updated, since these files may be moved to other locations. It is enough to mention that versions for all the VictoriaMetrics components must be updated.	2024-07-17 23:36:30 +02:00
Aliaksandr Valialkin	7e0fff224e	app/vmselect/vmui: run `make vmui-update` after `959a4383c5`	2024-07-17 23:09:25 +02:00
Aliaksandr Valialkin	47f62f2e75	docs/vmbackup.md: an attempt to fix formatting issue at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables	2024-07-17 23:01:32 +02:00
Aliaksandr Valialkin	009f71fc4c	docs/stream-aggregation.md: an attempt to fix yaml formatting at https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config	2024-07-17 22:43:54 +02:00
Aliaksandr Valialkin	c57c169dc4	deployment: update VictoriaMetrics Docker image from v1.101.0 to v1.102.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0	2024-07-17 22:35:00 +02:00
Aliaksandr Valialkin	1ebd4e8e43	docs/CHANGELOG.md: cut v1.102.0 release	2024-07-17 20:50:01 +02:00
Aliaksandr Valialkin	6eda5fd098	vendor: run `make vendor-update`	2024-07-17 20:47:33 +02:00
Aliaksandr Valialkin	dc6565c105	docs/CHANGELOG.md: consistently use new url format for the MetricsQL docs Use https://docs.victoriametrics.com/metricsql/ instead of https://docs.victoriametrics.com/MetricsQL.html . This removes unnecessary redirect from https://docs.victoriametrics.com/MetricsQL.html to https://docs.victoriametrics.com/metricsql/ This is a follow-up for `6a4bd5049b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547	2024-07-17 20:32:48 +02:00
Aliaksandr Valialkin	6dc2bcff2b	docs/CHANGELOG.md: document v1.97.6 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.6	2024-07-17 20:28:04 +02:00
Aliaksandr Valialkin	299c9af6de	docs/CHANGELOG.md: document v1.93.16 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.16	2024-07-17 19:48:15 +02:00
Aliaksandr Valialkin	65437f23fc	docs/CHANGELOG.md: order the changes at tip, so they are easier to read	2024-07-17 18:56:06 +02:00
Fred Navruzov	231d950764	docs/vmanomaly - v1.13.3 patch notes (#6659 ) ### Describe Your Changes Add patch note v1.13.3 to CHANGELOG doc page for `vmanomaly` ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-17 18:56:06 +02:00
Aliaksandr Valialkin	116b02eb29	docs/VictoriaLogs/querying/README.md: add link to Grafana datasource for VictoriaLogs to the top of the page	2024-07-17 18:34:47 +02:00
Aliaksandr Valialkin	f32e27e815	docs/Single-server-VictoriaMetrics.md: explain why images must have the same filepath as the corresponding docs at docs/ folder This is a follow-up for `f2c1d30011`	2024-07-17 18:26:46 +02:00
Aliaksandr Valialkin	326c9bc1aa	docs: fix spellcheck errors found by `make spellcheck` Thanks to the spellcheck Makefile rule added by @arkid15r at `fabf0b928e`	2024-07-17 18:21:17 +02:00
Aliaksandr Valialkin	9b529c2742	lib/backup/azremote: follow-up for `5fd3aef549` - Mention that credentials can be configured via env variables at both vmbackup and vmrestore docs. - Make clear that the AZURE_STORAGE_DOMAIN env var is optional at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables - Use string literals as is for env variable names instead of indirecting them via string constants. This makes easier to read and understand the code. These environment variable names aren't going to change in the future, so there is no sense in hiding them under string constants with some other names. - Refer to https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables in error messages when auth creds are improperly configured. This should simplify figuring out how to fix the error. - Simplify the code a bit at FS.newClient(), so it is easier to follow it now. While at it, remove the check when superflouos environment variables are set, since it is too fragile and it looks like it doesn't help properly configuring vmbackup / vmrestore. - Remove envLookuper indirection - just use 'func(name string) (string, bool)' type inline. This simplifies code reading and understanding. - Split TestFSInit() into TestFSInit_Failure() and TestFSInit_Success(). This simplifies the test code, so it should be easier to maintain in the future. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984	2024-07-17 17:55:39 +02:00
Aliaksandr Valialkin	97d696ae8b	all: substitute double "the the" with "the" This is a follow-up for `8786a08d27` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6600	2024-07-17 14:29:05 +02:00
Aliaksandr Valialkin	f8aa445945	all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter The %q formatter may result in incorrectly formatted JSON string if the original string contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string cannot be parsed by JSON parsers. This is a follow-up for `c0caa69939` See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24	2024-07-17 14:01:37 +02:00
Aliaksandr Valialkin	f2362812c3	lib/protoparser/graphite: use Regex.ReplaceAllLiteralString instead of Regex.ReplaceAllString for the case when the replacement cannot contain placeholders for capturing groups This is a follow-up for `74affa3aec`	2024-07-17 13:01:35 +02:00
Aliaksandr Valialkin	f7789b61e7	lib/protoparser/graphite: follow-up for `476faf5578` - Clarify the description of -graphite.sanitizeMetricName command-line flag at README.md - Do not sanitize tag values - only metric names and tag names must be sanitized, since they are treated specially by Grafana. Grafana doesn't apply any restrictions on tag values. - Properly replace more than two consecutive dots with a single dot. - Disallow unicode letters in metric names and tag names, since neither Prometheus nor Grafana do not support them. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6489 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077	2024-07-17 12:57:56 +02:00
Aliaksandr Valialkin	54d3abc092	lib: consistently use regexp.Regexp.ReplaceAllLiteralString instead of regexp.Regexp.ReplaceAllString in places where the replacement cannot contain matching group placeholders	2024-07-17 12:57:43 +02:00
rtm0	1b03d7e6de	Fix inconsistent error handling in Storage.AddRows() (#6583 ) `Storage.AddRows()` returns an error only in one case: when `Storage.updatePerDateData()` fails to unmarshal a `metricNameRaw`. But the same error is treated as a warning when it happens inside `Storage.add()` or returned by `Storage.prefillNextIndexDB()`. This commit fixes this inconsistency by treating the error returned by `Storage.updatePerDateData()` as a warning as well. As a result `Storage.add()` does not need a return value anymore and so doesn't `Storage.AddRows()`. Additionally, this commit adds a unit test that checks all cases that result in a row not being added to the storage. --------- Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-07-17 12:55:07 +02:00
Aliaksandr Valialkin	e1f6926d2a	docs/CONTRIBUTING.md: mention that docs/CHANGELOG.md shouldn't contain technical details for the changes The purpose of docs/CHANGELOG.md is to provide VictoriaMetrics users clear and concise information on what's changed at VictoriaMetrics components. Technical details of the change are unclear to most of VictoriaMetrics users, who are not familiar with VictoriaMetrics source code. These details complicate reading the docs/CHANGELOG.md by ordinary users, so do not clutter the changelog with technical details. If the user wants technical details, he can click the link to the related GitHub issue and/or pull request and dive into all the details he wants.	2024-07-17 11:51:08 +02:00
Aliaksandr Valialkin	3087749aa9	app/vmauth: properly handle the case when zero backend hosts are resolved at SRV DNS When zero backend hosts are resolved, then vmauth must return 'no backend hosts' error instead of crashing with panic This is a follow-up for `590aeccd7d` and `3a45bbb4e0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401	2024-07-17 11:34:33 +02:00
Aliaksandr Valialkin	31b8e9054d	app/vmauth: pool readTrackingBody structs in order to reduce pressure on Go GC - use pool for readTrackingBody structs in order to reduce pressure on Go GC - allow re-reading partially read request body - add missing tests for various cases of readTrackingBody usage This is a follow-up for `ad6af95183` and `4d66e042e3`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6446 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533	2024-07-17 11:34:32 +02:00
Aliaksandr Valialkin	d427a4cfaa	app/vmauth: use more clear names for the field and function added at `e666d64f1d` - Rename overrideHostHeader() function to hasEmptyHostHeader() - Rename overrideHostHeader field at UserInfo to useBackendHostHeader This should simplify the future maintenance of the code Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525	2024-07-17 11:34:32 +02:00
Aliaksandr Valialkin	111f7da946	Revert "app/vmauth: reader pool to reduce gc & mem alloc (#6533 )" This reverts commit `4d66e042e3`. Reasons for revert: - The commit makes unrelated invalid changes to docs/CHANGELOG.md - The changes at app/vmauth/main.go are too complex. It is better splitting them into two parts: - pooling readTrackingBody struct for reducing pressure on GC - avoiding to use readTrackingBody when -maxRequestBodySizeToRetry command-line flag is set to 0 Let's make this in the follow-up commits! Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533	2024-07-17 11:34:31 +02:00
Aliaksandr Valialkin	2acbd69a56	docs: remove the remaining mentioning about snap packages after the commit `0a42c8fd8b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543	2024-07-16 18:38:28 +02:00
Aliaksandr Valialkin	5168d0f754	lib/promrelabel: add test for IfExpression.String() function While at it, simplify this function a bit after the commit `861852f262` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462	2024-07-16 18:32:34 +02:00
Zakhar Bessarab	096abd827f	app/vmagent/kafka: fix non-unique metric naming (#774 ) * app/vmagent/kafka: fix non-unique metric naming Fix panic when using multiple topics with the same name. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/changelog: document bugfix Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/vmagent: more examples for Kafka ingestion with multiple brokers/topics groups Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-07-16 18:06:33 +02:00
Aliaksandr Valialkin	617a7b4db6	lib/promscrape/discovery/yandexcloud: follow-up for `070abe5c71` - Obtain IAM token via GCE-like API instead of Amazon EC2 IMDSv2 API, since it looks like IMDBSv2 API isn't supported by Yandex Cloud according to https://yandex.cloud/en/docs/security/standard/authentication#aws-token : > So far, Yandex Cloud does not support version 2, so it is strongly recommended > to technically disable getting a service account token via the Amazon EC2 metadata service. - Try obtaining IAM token via GCE-like API at first and then fall back to the deprecated Amazon EC2 IMDBSv1. This should prevent from auth errors for instances with disabled GCE-like auth API. This addresses @ITD27M01 concern at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513#issuecomment-1867794884 - Make more clear the description of the change at docs/CHANGELOG.md , add reference to the related issue. P.S. This change wasn't tested in prod because I have no access to Yandex Cloud. It is recommended to test this change by @ITD27M01 and @vmazgo , who filed the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6524	2024-07-16 18:06:33 +02:00
Alexander Marshalov	4881496d1a	docs: fixed image paths in the guide "Anomaly Detection and Alerting Setup" (#6651 )	2024-07-16 18:06:32 +02:00
f41gh7	146f69d6f8	docs: mention vmagent and vmgateway changes Signed-off-by: f41gh7 <nik@victoriametrics.com>	2024-07-16 18:02:50 +02:00
Aliaksandr Valialkin	0f56ab8774	docs/CHANGELOG.md: clarify docs and changelog after `e666d64f1d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6453 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525	2024-07-16 14:02:07 +02:00
Aliaksandr Valialkin	8e53212380	Revert "deployment: build image for vmagent streamaggr benchmark (#6515 )" This reverts commit `6b128da811`. Reason for revert: this complicates and slows down CI/CD without giving significant benefits in return. The idea of automatic building, publishing and deploying Docker images to our playground on every pull request and commit isn't very bright because of the following reasons: - It slows down CI/CD pipeline - It increases costs on CPU time spent at CI/CD pipeline - It contradicts goal #7 at https://docs.victoriametrics.com/goals/#goals and non-goal #8 at https://docs.victoriametrics.com/goals/#non-goals The previous workflow was much better - if we need to deploy some new Docker image at playground or staging environment, then just __manually__ build and deploy the needed Docker image there. If the manual process requires making too many steps, then think on how to automate these steps into a single Makefile command. Updates https://github.com/VictoriaMetrics/ops/pull/1297 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6515	2024-07-16 13:34:16 +02:00
Aliaksandr Valialkin	06f7f23ebe	vendor: update github.com/VictoriaMetrics/metrics from v1.35.0 to v1.35.1	2024-07-16 13:18:49 +02:00
Aliaksandr Valialkin	6d237da3f3	lib/promscrape: follow-up for `1e83598be3` - Clarify that the -promscrape.maxScrapeSize value is used for limiting the maximum scrape size if max_scrape_size option isn't set at https://docs.victoriametrics.com/sd_configs/#scrape_configs - Fix query example for scrape_response_size_bytes metric at https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics - Mention about max_scrape_size option at the -help description for -promscrape.maxScrapeSize command-line flag - Treat zero value for max_scrape_size option as 'no scrape size limit' - Change float64 to int type for scrapeResponseSize struct fields and function args, since response size cannot be fractional - Optimize isAutoMetric() function a bit - Sort auto metrics in alphabetical order in isAutoMetric() and in scrapeWork.addAutoMetrics() functions for better maintainability in the future Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6434 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429	2024-07-16 12:38:41 +02:00
Github Actions	081bf0cd3e	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3cbd5e9 (#6649 )	2024-07-16 12:38:40 +02:00
Aliaksandr Valialkin	6a1ea788b9	Revert "docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494 )" This reverts commit `b37b288dce`. Reason for revert: statsd protocol support has been reverted - see `2da7dfc754`	2024-07-16 10:59:18 +02:00
Aliaksandr Valialkin	8b1c38abde	app/vmauth: follow-up for `3a45bbb4e0` - Move the test for SRV discovery into a separate function. This allows verifying round-robin discovery across SRV records. - Restore the original netutil.Resolver after the test finishes, so it doesn't interfere with other tests. - Move the description of the bugfix into the correct place at docs/CHANGELOG.md - it should be placed under v1.102.0-rc2 instead of v1.102.0-rc1. - Remove unneeded code in URLPrefix.sanitizeAndInitialize(), since it is expected this function is called only once for finishing URLPrefix initializiation. In this case URLPrefix.nextDiscoveryDeadline and URLPrefix.n are equal to 0 according to https://pkg.go.dev/sync/atomic#Uint64 - Properly fix the bug at URLPrefix.discoverBackendAddrsIfNeeded() - it is expected that hostToAddrs map uses the original hostname keys, including 'srv+' prefix, so it shouldn't be removed when looping over up.busOriginal. Instead, the 'srv+' prefix must be removed from the hostname only locally before passing the hostname to netutil.Resolver.LookupSRV. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401	2024-07-16 10:41:08 +02:00
Artem Navoiev	36955ff066	docs/data-ingestion use img tag instead of markdown image. this allow docs that sync automation to correctly render images Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-16 10:41:06 +02:00
Aliaksandr Valialkin	893a555051	Revert "lib/protoparser/opentelemetry/firehose: escape requestID before returning it to user (#6451 )" This reverts commit `cd1aca217c`. Reason for revert: this commit has no sense, since the firehose response has application/json content-type, so it must contain JSON-encoded timestamp and requestId fields according to https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat . HTML-escaping the requestId field may break the response, so the client couldn't correctly recognize the html-escaped requestId. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6451	2024-07-16 09:50:16 +02:00
Aliaksandr Valialkin	468c04d3c2	app/vmauth: clarify the description for -idleConnTimeout command-line flag This is a follow-up for `d44058bcd6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6388	2024-07-16 09:40:01 +02:00
Aliaksandr Valialkin	8b76a40715	lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag() This is a follow-up for `61dce6f2a1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6338 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329	2024-07-16 01:08:41 +02:00
Aliaksandr Valialkin	0cf18a6f63	lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now This makes unnecessary the checkDeleted variable at lib/storage/index_db.go This is a follow-up for `b984f4672e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6342	2024-07-16 00:00:46 +02:00
Aliaksandr Valialkin	cbbc6e7141	lib/mergeset: properly update TableMetrics.TooLongItemsDroppedTotal inside Table.UpdateMetrics Substitute '+=' with '=', since tooLongItemsTotal is global counter, which doesn't belong to the Table struct. This is a follow-up for `69d244e6fb` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6297	2024-07-15 23:41:26 +02:00
Aliaksandr Valialkin	aa52d6cd9b	app/vminsert: increase default value for -maxLabelValueLen command-line flag from 1KiB to 4KiB It has been appeared that the standard Kubernetes monitoring can generate labels with sizes up to 4KiB This is a follow-up for `a5d1013042` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176	2024-07-15 23:32:54 +02:00
Aliaksandr Valialkin	476bf400ac	lib/{httputils,netutil}: move httputils.GetStatDialFunc to netutil.NewStatDialFunc - Rename GetStatDialFunc to NewStatDialFunc, since it returns new function with every call - NewStatDialFunc isn't related to http in any way, so it must be moved from lib/httputils to lib/netutil - Simplify the implementation of NewStatDialFunc by removing sync.Map from there. - Use netutil.NewStatDialFunc at app/vmauth and lib/promscrape/discoveryutils - Use gauge instead of counter type for *_conns metric This is a follow-up for `d7b5062917` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6299	2024-07-15 23:05:46 +02:00
Aliaksandr Valialkin	f4dce57ebe	lib/streamaggr/streamaggr.go: typo fix after `5e29ef5ed5`: IgnoredNaNSamples -> ignoredNaNSamples	2024-07-15 21:59:03 +02:00
Aliaksandr Valialkin	a18eb2dacd	docs/CHANGELOG.md: typo fix: vl_streamaggr -> vm_streamaggr Thanks to @AndrewChubatiuk for the comment at `db557b86ee (r144259373)`	2024-07-15 21:50:35 +02:00
Aliaksandr Valialkin	fec9e9a0a7	docs/stream-aggregation.md: clarify "Routing" chapter a bit after `f153f54d11`	2024-07-15 21:42:56 +02:00
Aliaksandr Valialkin	353766061b	app/{vminsert,vmselect}: pass proper args to metrics.UnregisterSet() after `a8356f3a26`	2024-07-15 20:27:40 +02:00
Aliaksandr Valialkin	cbc637d1dd	app/vmagent/remotewrite: follow-up for `f153f54d11` - Move the remaining code responsible for stream aggregation initialization from remotewrite.go to streamaggr.go . This improves code maintainability a bit. - Properly shut down streamaggr.Aggregators initialized inside remotewrite.CheckStreamAggrConfigs(). This prevents from potential resource leaks. - Use separate functions for initializing and reloading of global stream aggregation and per-remoteWrite.url stream aggregation. This makes the code easier to read and maintain. This also fixes INFO and ERROR logs emitted by these functions. - Add an ability to specify `name` option in every stream aggregation config. This option is used as `name` label in metrics exposed by stream aggregation at /metrics page. This simplifies investigation of the exposed metrics. - Add `path` label additionally to `name`, `url` and `position` labels at metrics exposed by streaming aggregation. This label should simplify investigation of the exposed metrics. - Remove `match` and `group` labels from metrics exposed by streaming aggregation, since they have little practical applicability: it is hard to use these labels in query filters and aggregation functions. - Rename the metric `vm_streamaggr_flushed_samples_total` to less misleading `vm_streamaggr_output_samples_total` . This metric shows the number of samples generated by the corresponding streaming aggregation rule. This metric has been added in the commit `861852f262` . See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462 - Remove the metric `vm_streamaggr_stale_samples_total`, since it is unclear how it can be used in practice. This metric has been added in the commit `861852f262` . See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462 - Remove Alias and aggrID fields from streamaggr.Options struct, since these fields aren't related to optional params, which could modify the behaviour of the constructed streaming aggregator. Convert the Alias field to regular argument passed to LoadFromFile() function, since this argument is mandatory. - Pass Options arg to LoadFromFile() function by reference, since this structure is quite big. This also allows passing nil instead of Options when default options are enough. - Add `name`, `path`, `url` and `position` labels to `vm_streamaggr_dedup_state_size_bytes` and `vm_streamaggr_dedup_state_items_count` metrics, so they have consistent set of labels comparing to the rest of streaming aggregation metrics. - Convert aggregator.aggrStates field type from `map[string]aggrState` to `[]aggrOutput`, where `aggrOutput` contains the corresponding `aggrState` plus all the related metrics (currently only `vm_streamaggr_output_samples_total` metric is exposed with the corresponding `output` label per each configured output function). This simplifies and speeds up the code responsible for updating per-output metrics. This is a follow-up for the commit `2eb1bc4f81` . See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6604 - Added missing urls to docs ( https://docs.victoriametrics.com/stream-aggregation/ ) in error messages. These urls help users figuring out why VictoriaMetrics or vmagent generates the corresponding error messages. The urls were removed for unknown reason in the commit `2eb1bc4f81` . - Fix incorrect update for `vm_streamaggr_output_samples_total` metric in flushCtx.appendSeriesWithExtraLabel() function. While at it, reduce memory usage by limiting the maximum number of samples per flush to 10K. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6268	2024-07-15 20:25:36 +02:00
Artem Navoiev	ad90daf663	remove coc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-15 20:25:15 +02:00
Mathias Palmersheim	22f269ad0f	Added docs for telegraf proxmox and vector (#6619 ) ### Describe Your Changes initial docs to implement #6618 more platforms can be added on this branch or on future commits. ### Checklist The following checks are mandatory: - [x ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-15 20:25:15 +02:00
Fred Navruzov	6278c6ddf2	docs/vmanomaly - v1.13.2 updates (#6646 ) ### Describe Your Changes Doc updates after v1.13.2 release of `vmanomaly` ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-15 20:25:14 +02:00
Artem Navoiev	6603b52159	docs/VictoriaLogs link Github Issues with Roadmap items. Add informat… (#6633 ) …ion about Grafana Datasource in quering ### Describe Your Changes Update Roadmap and Querying documentation for VictoriaLogs ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-15 20:25:14 +02:00
Aliaksandr Valialkin	a8356f3a26	vendor: update github.com/VictoriaMetrics/metrics from v1.34.1 to v1.35.0 Fix potential memory leaks across VictoriaMetrics codebase after metrics.UnregisterSet(s) call because of missing s.UnregisterAllMetrics() call. This is a follow-up for `6a6e34ab8e` . It is OK if some vmauth metrics aren't visible for a few microseconds when the previous metrics are unregistered and new metrics weren't registered yet. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6252 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5805	2024-07-15 10:45:39 +02:00
Aliaksandr Valialkin	4878152678	lib/{storage,mergeset}: do not allow setting dataFlushInterval to values smaller than pending{Items,Rows}FlushInterval Pending rows and items unconditionally remain in memory for up to pending{Items,Rows}FlushInterval, so there is no any sense in setting dataFlushInterval (the interval for guaranteed flush of in-memory data to disk) to values smaller than pending{Items,Rows}FlushInterval, since this doesn't affect the interval for flushing pending rows and items from memory to disk. This is a follow-up for `4c80b17027` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6221	2024-07-15 10:11:23 +02:00
Aliaksandr Valialkin	c3d2351948	lib/streamaggr: consistently use alphabetical order of benchmarked stream aggregation outputs	2024-07-15 09:53:26 +02:00
Aliaksandr Valialkin	3365dd508f	app/vmagent/remotewrite: do not spend CPU time on an attempt to send data to blocked queue if some queues are unblocked Previously remotewrite.TryPush() was trying to send data to remote storages with blocked persistent queues, if some persistent queues to other remote storage systems were unblocked. This resulted in excess CPU usage on relabeling and stream aggregation for the remote storage with blocked queues. The solution is to check whether some peristent storages have blocked queues and skip them before applying per- -remoteWrite.url relabeling and streaming aggregation. While at it, properly update per- -remoteWrite.url vmagent_remotewrite_samples_dropped_total and vmagent_remotewrite_push_failures_total counters when global streaming aggregation cannot send data to remote storage systems because of blocked queues. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467 and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6268 . This is a follow-up for `87fd400dfc` and `f153f54d11` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6248 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065	2024-07-15 09:40:34 +02:00
Aliaksandr Valialkin	21f049e211	lib/streamaggr: follow-up for `9c3d44c8c9` - Consistently enumerate stream aggregation outputs in alphabetical order across the source code and docs. This should simplify future maintenance of the corresponding code and docs. - Fix the link to `rate_sum()` at `see also` section of `rate_avg()` docs. - Make more clear the docs for `rate_sum()` and `rate_avg()` outputs. - Encapsulate output metric suffix inside rateAggrState. This eliminates possible bugs related to incorrect suffix passing to newRateAggrState(). - Rename rateAggrState.total field to less misleading rateAggrState.increase name, since it calculates counter increase in the current aggregation window. - Set rateLastValueState.prevTimestamp on the first sample in time series instead of the second sample. This makes more clear the code logic. - Move the code for removing outdated entries at rateAggrState into removeOldEntries() function. This make the code logic inside rateAggrState.flushState() more clear. - Do not write output sample with zero value if there are no input series, which could be used for calculating the rate, e.g. if only a single sample is registered for every input series. - Do not take into account input series with a single registered sample when calculating rate_avg(), since this leads to incorrect results. - Move {rate,total}AggrState.flushState() function to the end of rate.go and total.go files, so they look more similar. This shuld simplify future mantenance. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6243	2024-07-15 08:44:48 +02:00
Aliaksandr Valialkin	4921ec5604	docs/CHANGELOG.md: use new link to VictoriaMetrics cluster docs instead of old link The old link was changed globally to the new link in the commit `f4b1cbfef0` . Unfortunately, old links are still posted in new commits :( This is a follow-up for `680b8c25c8` . While at it, remove duplicate 'len(*remoteWriteURLs) > 0' check in the remotewrite.Init() functions, since this check is already made at the beginning of the function. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6253	2024-07-13 03:04:20 +02:00
Aliaksandr Valialkin	8188766526	docs/CHANGELOG.md: consistently use new url to vmagent docs - https://docs.victoriametrics.com/vmagent/ - instead of old one - https://docs.victoriametrics.com/vmagent.html See the previous commit, which was making the same thing a few months ago - `c81a633b02` Unfortunately, new commits continue using old links :(	2024-07-13 02:40:06 +02:00
Aliaksandr Valialkin	bc1f92d7f5	app/vmagent/remotewrite: follow-up for `87fd400dfc` - Drop samples and return true from remotewrite.TryPush() at fast path when all the remote storage systems are configured with the disabled on-disk queue, every in-memory queue is full and -remoteWrite.dropSamplesOnOverload is set to true. This case is quite common, so it should be optimized. Previously additional CPU time was spent on per-remoteWriteCtx relabeling and other processing in this case. - Properly count the number of dropped samples inside remoteWriteCtx.pushInternalTrackDropped(). Previously dropped samples were counted only if -remoteWrite.dropSamplesOnOverload flag is set. In reality, the samples are dropped when they couldn't be sent to the queue because in-memory queue is full and on-disk queue is disabled. The remoteWriteCtx.pushInternalTrackDropped() function is called by streaming aggregation for pushing the aggregated data to the remote storage. Streaming aggregation cannot wait until the remote storage processes pending data, so it drops aggregated samples in this case. - Clarify the description for -remoteWrite.disableOnDiskQueue command-line flag at -help output, so it is clear that this flag can be set individually per each -remoteWrite.url. - Make the -remoteWrite.dropSamplesOnOverload flag global. If some of the remote storage systems are configured with the disabled on-disk queue, then there is no sense in keeping samples on some of these systems, while dropping samples on the remaining systems, since this will result in global stall on the remote storage system with the disabled on-disk queue and with the -remoteWrite.dropSamplesOnOverload=false flag. vmagent will always return false from remotewrite.TryPush() in this case. This will result in infinite duplicate samples written to the remaining remote storage systems. That's why the -remoteWrite.dropSamplesOnOverload is forcibly set to true if more than one -remoteWrite.disableOnDiskQueue flag is set. This allows proceeding with newly scraped / pushed samples by sending them to the remaining remote storage systems, while dropping them on overloaded systems with the -remoteWrite.disableOnDiskQueue flag set. - Verify that the remoteWriteCtx.TryPush() returns true in the TestRemoteWriteContext_TryPush_ImmutableTimeseries test. - Mention in vmagent docs that the -remoteWrite.disableOnDiskQueue command-line flag can be set individually per each -remoteWrite.url. See https://docs.victoriametrics.com/vmagent/#disabling-on-disk-persistence Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6248 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065	2024-07-13 02:30:10 +02:00
Aliaksandr Valialkin	5c7345b8ce	app/victoria-logs/Makefile: add `make victoria-logs-linux-loong64` build rule This is a follow-up for `80f3644ee3` The https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6222 missed build rule for VictoriaLogs.	2024-07-12 23:13:19 +02:00
Aliaksandr Valialkin	43fc1183b9	app/vmalert: switch from table-driven tests to f-tests This makes test code more clear and reduces the number of code lines by 500. This also simplifies debugging tests. See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e While at it, consistently use t.Fatal* instead of t.Error* across tests, since t.Error* requires more boilerplate code, which can result in additional bugs inside tests. While t.Error* allows writing logging errors for the same, this doesn't simplify fixing broken tests most of the time. This is a follow-up for `a9525da8a4`	2024-07-12 22:45:50 +02:00
Aliaksandr Valialkin	04a304fd39	app/vmctl: switch from table-driven tests to f-tests This simplifies debugging tests and makes the test code more clear and concise. See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e While at is, consistently use t.Fatal* instead of t.Error* across tests, since t.Error* requires more boilerplate code, which can result in additional bugs inside tests. While t.Error* allows writing logging errors for the same, this doesn't simplify fixing broken tests most of the time. This is a follow-up for `a9525da8a4`	2024-07-12 22:45:49 +02:00
Alexander Marshalov	a2fca24402	vmcloud: fixed images in cloud alertmanager docs (#6641 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-12 22:45:26 +02:00
Alexander Marshalov	96306885ae	vmcloud: fixed image paths for cloud alertmanager docs (#6640 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-12 22:45:26 +02:00
Alexander Marshalov	eeb8717c66	vmcloud: updated docs about using alertmanager (#6638 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-07-12 22:45:25 +02:00
hagen1778	f7f684fa0d	docs: mention stream aggregation in churn rate section Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `fbdaba4f8c`)	2024-07-12 14:19:28 +02:00
hagen1778	feb58e70f4	docs: mention assets naming policy and placement Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f2c1d30011`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-12 14:19:27 +02:00
hagen1778	c835a6351e	lib/streamaggr: add missing test cases Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2f65956259`)	2024-07-12 14:19:17 +02:00
Hui Wang	f3cbd62823	vmagent: fix `vm_streamaggr_flushed_samples_total` counter (#6604 ) We use `vm_streamaggr_flushed_samples_total` to show the number of produced samples by aggregation rule, previously it was overcounted, and doesn't account for `output_relabel_configs`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2eb1bc4f81`)	2024-07-12 14:19:17 +02:00
Aliaksandr Valialkin	7c97cef95c	app: consistently use t.Fatal* instead of t.Error* (except of app/vmalert and app/vmctl - these packages will be processed in a separate commit) Consistently using t.Fatal* simplifies the test code and makes it less fragile, since it is common error to forget to make proper cleanup after t.Error* call. Also t.Error* calls do not provide any practical benefits when some tests fail. They just clutter test output with additional noise information, which do not help in fixing failing tests most of the time. This is a follow-up for `a9525da8a4`	2024-07-11 16:01:25 +02:00
Arkadii Yakovets	8645b2cc8e	docs: add spellcheck command (#6562 ) ### Describe Your Changes Implement spellcheck command: - add cspell configuration files - dockerize spellchecking process - add Makefile targets This PR adds a standalone `make spellcheck` target to check `docs/.md` files for spelling errors. The target process is dockerized to be run in a separate npm environment. Some `docs/` typo fixes also included. ### Checklist The following checks are mandatory*: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `fabf0b928e`)	2024-07-11 12:40:24 +02:00
Zhu Jiekun	2ea575e776	vmalert: [bug] fixed System hyperlink 404 redirect (#6620 ) ### Describe Your Changes As mentioned in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6603, some hyperlinks under `vmalert` -> `System` section is not working as expected. Pages and redirection: - For page `http://127.0.0.1:8880/`: `flags` button will redirect to `http://127.0.0.1:8880/flags` - For page `http://127.0.0.1:8880/vmalert`: `http://127.0.0.1:8880/flags` - For page `http://127.0.0.1:8880/vmalert/`: `http://127.0.0.1:8880/vmalert/flags` (page not exists) - Similar redirection could be observed with `-http.pathPrefix` Two potential ways to avoid 404 redirection: 1. avoid visiting `/vmalert/` (I'm trying to do this). 2. provide support for `/vmalert/flags`. `/vmalert/` could be visit only when user click other navigator (e.g. Group) and click vmalert again: ![Peek 2024-07-10 10-07](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/13d7b147-a1b6-4e93-9ee0-26f881a16bef) Because: `http://127.0.0.1:8880/vmalert/groups?search=` + `<a class="nav-link" href=".">` = `http://127.0.0.1:8880/vmalert/` So I'm trying to change the `href="."` to `href="../vmalert"`. ### Checklist The following checks are mandatory: - [X] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `cadf1eb5ab`)	2024-07-11 12:40:23 +02:00
Artem Navoiev	38bb2372a4	docs: quick use official names for victoriametrics components (#6626 ) ### Describe Your Changes Replace VM- with VictoriaMetrics in QuickStart Keep the previous anchors for backward compatibility ### Checklist The following checks are mandatory: - [ x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `5b43675af6`)	2024-07-11 12:40:23 +02:00
Nikolay	d136bfcdbf	make vendor-update (#6627 ) make vendor-update	2024-07-10 17:17:08 +02:00
Github Actions	144843a1b8	Automatic update operator docs from VictoriaMetrics/operator@4774786 (#6617 ) Automated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit `c688b53f56`)	2024-07-10 17:11:15 +02:00
Zakhar Bessarab	401ae72587	app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547 ) ### Describe Your Changes In most cases histograms are exposed in sorted manner with lower buckets being first. This means that during scraping buckets with lower bounds have higher chance of being updated earlier than upper ones. Previously, values were propagated from upper to lower bounds, which means that in most cases that would produce results higher than expected once all buckets will become updated. Propagating from upper bound effectively limits highest value of histogram to the value of previous scrape. Once the data will become consistent in the subsequent evaluation this causes spikes in the result. Changing propagation to be from lower to higher buckets reduces value spikes in most cases due to nature of the original inconsistency. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580 An example histogram with previous(red) and updated(blue) versions: ![1719565540](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/1367798/605c5e60-6abe-45b5-89b2-d470b60127b8) This also makes logic of filling nan values with lower buckets values: [1 2 3 nan nan nan] => [1 2 3 3 3 3] obsolete. Since buckets are now fixed from lower ones to upper this happens in the main loop, so there is no need in a second one. --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6a4bd5049b`)	2024-07-10 15:17:08 +02:00
hagen1778	7370f84b97	lib/bakcup/azremote: follow-up after `5fd3aef549` Simplify tests by converting them to f-tests. `5fd3aef549` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `03e4c5c19c`)	2024-07-10 15:17:06 +02:00
hagen1778	db9a1bb59e	docs: follow-up after `c341369fc1` * account for `source` tag in `docs-sync` command * run `make docs-sync` `c341369fc1` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1c155a0cd1`)	2024-07-10 12:26:26 +02:00
hagen1778	64a896e535	docs/vmrestore: link to auth examples in vmbackup vmbackup, vmrestore and vmbackupmanager use the same libs for integrations with object storage. That means the auth can be configured in the same way for all of them. So the docs should have either identical config section for all 3 components, or we should cross-link to one source of truth. This change removes incomplete auth options from vmrestore docs and adds link to complete auth options in vmbackup instead. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b29154a497`)	2024-07-10 12:26:23 +02:00
justinrush	e65e55e2dd	lib/backup: add support for Azure Managed Identity (#6518 ) ### Describe Your Changes These changes support using Azure Managed Identity for the `vmbackup` utility. It adds two new environment variables: * `AZURE_USE_DEFAULT_CREDENTIAL`: Instructs the `vmbackup` utility to build a connection using the [Azure Default Credential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity@v1.5.2#NewDefaultAzureCredential) mode. This causes the Azure SDK to check for a variety of environment variables to try and make a connection. By default, it tries to use managed identity if that is set up. This will close https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984 ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). ### Testing However you normally test the `vmbackup` utility using Azure Blob should continue to work without any changes. The set up for that is environment specific and not listed out here. Once regression testing has been done you can set up [Azure Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) so your resource (AKS, VM, etc), can use that credential method. Once it is set up, update your environment variables according to the updated documentation. I added unit tests to the `FS.Init` function, then made my changes, then updated the unit tests to capture the new branches. I tested this in our environment, but with SAS token auth and managed identity and it works as expected. --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Justin Rush <jarush@epic.com> Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5fd3aef549`)	2024-07-10 12:26:21 +02:00
Aliaksandr Valialkin	6b4c510091	docs/VictoriaLogs/CHANGELOG.md: cut v0.28.0-victorialogs release	2024-07-10 03:08:26 +02:00
Aliaksandr Valialkin	a1decb5ca1	app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages	2024-07-10 03:05:55 +02:00
Aliaksandr Valialkin	32ae40410c	app/vlselect/vmui: run `make vmui-logs-update` after `662e026279`	2024-07-10 03:05:55 +02:00
Aliaksandr Valialkin	b8a8d3d6f1	lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API	2024-07-10 00:39:16 +02:00
Aliaksandr Valialkin	d6415b2572	all: consistently use 'any' instead of 'interface{}' 'any' type is supported starting from Go1.18. Let's consistently use it instead of 'interface{}' type across the code base, since `any` is easier to read than 'interface{}'.	2024-07-10 00:23:26 +02:00
Aliaksandr Valialkin	73ca22bb7d	app/vlinsert/loki: remove unused functions from the generated protobuf code	2024-07-10 00:22:10 +02:00
Aliaksandr Valialkin	9edeecabc8	lib: consistently use f-tests instead of table-driven tests This makes easier to read and debug these tests. This also reduces test lines count by 15% from 3K to 2.5K . See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e . While at it, consistently use t.Fatal* instead of t.Error, since t.Error usually leads to more complicated and fragile tests, while it doesn't bring any practical benefits over t.Fatal*.	2024-07-09 22:39:13 +02:00
Yury Molodov	33bd5ccbab	vmui/logs: add spinner to bar chart (#6577 ) Add a spinner to the bar chart https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `662e026279`)	2024-07-09 18:27:23 +02:00
Hui Wang	6f602a4ef5	security: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1 See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html >including security fixes for: OPENSSL [CVE-2024-4741](https://security.alpinelinux.org/vuln/CVE-2024-4741) BUSYBOX [CVE-2023-42364](https://security.alpinelinux.org/vuln/CVE-2023-42364) [CVE-2023-42365](https://security.alpinelinux.org/vuln/CVE-2023-42365) (cherry picked from commit `8e9f98e725`)	2024-07-09 11:38:44 +02:00
Phuong Le	a0920d2f27	docs: use white version of logo for dark theme (#6610 ) The logo on the README is hard to recognize in dark mode on GitHub, so I made it responsive based on user preferences. (cherry picked from commit `c341369fc1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-09 11:20:40 +02:00
hagen1778	f33f7d60ed	docs: follow-up `0e1dbdee28` * restore old anchor names to keep links compatibility. See https://docs.victoriametrics.com/#documentation requirements * consistently use the same format for commands `sh` as it makes it better renderred and automatically adds `copy` button to fileds with commands * simplify the text by removing extra points in the list * add recommendations for installing the cluster setup * explicitly mention the ports services are listening on * add description for `storageNode` cmd-line flag to inform the reader what values need to be put into it * fix the incorrect vmui link in cluster installation recommendation * rename component anchors to be more unique, because URL doesn't respect hierarchy for the anchored links and may result into conflicts in future Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6e17255ec0`)	2024-07-09 10:53:07 +02:00
Artem Navoiev	b73214cb35	add alt to cluster logo, this allows to hide it on our docs site Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `878c727a3a`)	2024-07-09 10:53:04 +02:00
Mathias Palmersheim	3ec15e9ba8	Added Binary Install Docs (#6598 ) ### Describe Your Changes Updated Quickstart guide for VIctoriaMetrics and VictoriaMetrics Cluster to include instructions for installing the binaries by hand ### Checklist The following checks are mandatory: - [ x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `0e1dbdee28`)	2024-07-09 10:53:00 +02:00
Github Actions	3eb8068b77	Automatic update operator docs from VictoriaMetrics/operator@9ef3d95 (#6596 ) Automated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit `f41f5ef62e`)	2024-07-09 10:52:57 +02:00
Daria Karavaieva	0897887d07	docs/vmanomaly:custom model guide fix (#6594 ) ### Describe Your Changes Fixed Custom Model guide according to newer `vmanomaly` versions ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `7478d2de4c`)	2024-07-09 10:52:55 +02:00
Github Actions	5fd4edbf8c	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@e77b503 (#6601 ) (cherry picked from commit `108a5e10d8`)	2024-07-09 10:52:53 +02:00
Artem Navoiev	7b508a9334	fix typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `4527020a68`)	2024-07-09 10:52:50 +02:00
omahs	efc6b00b2c	docs: fix typos (#6600 ) ### Describe Your Changes docs: fix typos ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `8786a08d27`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-09 10:52:47 +02:00
Aliaksandr Valialkin	f3be3573e7	lib/promscrape/discovery/vultr: follow-up after `17e3d019d2` - Sort the discovered labels in alphabetical order at https://docs.victoriametrics.com/sd_configs/#vultr_sd_configs - Rename VultrConfigs to VultrSDConfigs to be consistent with the naming for other SD configs. - Prepare query arg filters for `list instances API` at newAPIConfig() instead of passing them in a separate listParams struct. This simplifies the code a bit. - Return error when bearer token isn't set at vultr_sd_configs, since this token is mandatory according to https://docs.victoriametrics.com/sd_configs/#vultr_sd_configs - Remove unused fields from the parsed response from Vultr list instances API in order to simplify the code a bit. - Remove double logging of errors inside getInstances() function, since these errors must be already logged by the caller. - Simplify tests, so they are easier to maintain. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6068	2024-07-05 17:40:39 +02:00
Hui Wang	e031712f21	vmagent-dashboard: update streaming aggregation panels (#6588 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `c1c2286e09`)	2024-07-05 15:16:19 +02:00
Github Actions	67a7cf37c2	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@0c2f8ef (#6592 ) (cherry picked from commit `5275b36380`)	2024-07-05 15:16:14 +02:00
Arkadii Yakovets	36311825cc	Optimize CI workflows (#6551 ) ### Describe Your Changes This PR is aimed to change the currently in place configuration of running Go related jobs for code changes that don't contain actual Go files ([example 1](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6517/checks) - 2m32s , [example 2](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543/checks) - 4m11s). In order to do that the `build` workflow was extracted from Go related workflow (now it doesn't require lint as a `need` step -- let me know if it's something we want to keep). It will run upon the same triggers as before the change. The `main` workflow now will be triggered by `.go` pattern only and contains lint/test steps that are relevant for Go file changes. I expect this PR + https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6540 to improve CI minutes usage. ### Checklist The following checks are mandatory**: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> (cherry picked from commit `c14e827cc5`)	2024-07-05 15:14:03 +02:00
Artem Navoiev	0d7bed212f	docs: remove templates Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `d5e4857a27`)	2024-07-05 15:14:01 +02:00
Github Actions	6178e48e01	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@e0a0b2d (#6591 ) (cherry picked from commit `fe42884b1b`)	2024-07-05 15:13:56 +02:00
hagen1778	819f1da12b	docs: clarify multitenancy via labels for vmagent Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4d92f875fd`)	2024-07-05 13:58:22 +02:00
hagen1778	baddb76b4b	deployment: fix typo after `cb19335a9f` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3d164d7e31`)	2024-07-05 13:58:21 +02:00
Github Actions	6f12ca71eb	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@47c26c4 (#6586 ) (cherry picked from commit `7e781f0f78`)	2024-07-05 09:49:32 +02:00
hagen1778	7d98c200eb	docs: update refereneces to victoriametrics-datasource plugin The plugin was renamed in https://github.com/VictoriaMetrics/victoriametrics-datasource/pull/178 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ee66fb4387`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-05 09:49:28 +02:00
Yury Molodov	7fc9912d15	vmui: add compact JSON display (#6582 ) ### Describe Your Changes If a JSON element has only one field, it will be displayed on a single line. #6559 \| Old Display \| New Display \| \|-------------\|-------------\| \| ![image](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/8866517b-a49d-450f-904c-19117397a078) \| ![image](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/8e222b43-a4cb-4f32-9a79-6199778404d3) \| ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `959a4383c5`)	2024-07-05 09:49:12 +02:00
Github Actions	9e324cc28a	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@b326811 (#6580 ) (cherry picked from commit `7e169a767d`)	2024-07-05 09:49:10 +02:00
Github Actions	8df41f37e6	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@4ed20bd (#6578 ) (cherry picked from commit `11989db586`)	2024-07-05 09:49:08 +02:00
Hui Wang	bbd49a1a61	vmalert: allow omitting `-replay.timeTo` in replay mode, default valu… (#6575 ) …e is the current timestamp address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6492 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3169524fb7`)	2024-07-05 09:49:06 +02:00
Roman Khavronenko	b13c363f12	app/vmalert: add examples for `source` override (#6561 ) The change adds a new docs section with examples on how source can be overridden. It should address questions like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6536 While there, fix the example in `external.alert.source` cmd-line flag and docker-compose examples. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c429bbf889`)	2024-07-05 09:49:03 +02:00
Aliaksandr Valialkin	210e5b8957	docs/VictoriaLogs/CHANGELOG.md: cut v0.27.1-victorialogs	2024-07-05 01:22:50 +02:00
Aliaksandr Valialkin	6397c38a0a	lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b . See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24	2024-07-05 01:22:49 +02:00
Aliaksandr Valialkin	73d286847b	vendor: update github.com/valyala/quicktemplate from v1.7.0 to v1.8.0	2024-07-05 01:22:49 +02:00
Github Actions	3d6e3ee98f	Automatic update operator docs from VictoriaMetrics/operator@0c311bb (#6574 )	2024-07-04 11:46:16 +02:00
Aliaksandr Valialkin	172ae1adf7	Revert `c6c5a5a186` and `b2765c45d0` Reason for revert: There are many statsd servers exist: - https://github.com/statsd/statsd - classical statsd server - https://docs.datadoghq.com/developers/dogstatsd/ - statsd server from DataDog built into DatDog Agent ( https://docs.datadoghq.com/agent/ ) - https://github.com/avito-tech/bioyino - high-performance statsd server - https://github.com/atlassian/gostatsd - statsd server in Go - https://github.com/prometheus/statsd_exporter - statsd server, which exposes the aggregated data as Prometheus metrics These servers can be used for efficient aggregating of statsd data and sending it to VictoriaMetrics according to https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd ( the https://github.com/prometheus/statsd_exporter can be scraped as usual Prometheus target according to https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter ). Adding support for statsd data ingestion protocol into VictoriaMetrics makes sense only if it provides significant advantages over the existing statsd servers, while has no significant drawbacks comparing to existing statsd servers. The main advantage of statsd server built into VictoriaMetrics and vmagent - getting rid of additional statsd server. The main drawback is non-trivial and inconvenient streaming aggregation configs, which must be used for the ingested statsd metrics ( see https://docs.victoriametrics.com/stream-aggregation/ ). These configs are incompatible with the configs for standalone statsd servers. So you need to manually translate configs of the used statsd server to stream aggregation configs when migrating from standalone statsd server to statsd server built into VictoriaMetrics (or vmagent). Another important drawback is that it is very easy to shoot yourself in the foot when using built-in statsd server with the -statsd.disableAggregationEnforcement command-line flag or with improperly configured streaming aggregation. In this case the ingested statsd metrics will be stored to VictoriaMetrics as is without any aggregation. This may result in high CPU usage during data ingestion, high disk space usage for storing all the unaggregated statsd metrics and high CPU usage during querying, since all the unaggregated metrics must be read, unpacked and processed during querying. P.S. Built-in statsd server can be added to VictoriaMetrics and vmagent after figuring out more ergonomic specialized configuration for aggregating of statsd metrics. The main requirements for this configuration: - easy to write, read and update (ideally it should work out of the box for most cases without additional configuration) - hard to misconfigure (e.g. hard to shoot yourself in the foot) It would be great if this configuration will be compatible with the configuration of the most widely used statsd server. In the mean time it is recommended continue using external statsd server. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6265 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5053 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5052 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4600	2024-07-03 23:57:49 +02:00
Aliaksandr Valialkin	7a60e8abf7	lib/promscrape: use prompbmarshal.MustParsePromMetrics function at parseData() test function The prompbmarshal.MustParsePromMetrics function has been added in the commit `cc4d57d650`	2024-07-03 16:10:37 +02:00
Aliaksandr Valialkin	cd152693c6	Revert "Exemplar support (#5982 )" This reverts commit `5a3abfa041`. Reason for revert: exemplars aren't in wide use because they have numerous issues which prevent their adoption (see below). Adding support for examplars into VictoriaMetrics introduces non-trivial code changes. These code changes need to be supported forever once the release of VictoriaMetrics with exemplar support is published. That's why I don't think this is a good feature despite that the source code of the reverted commit has an excellent quality. See https://docs.victoriametrics.com/goals/ . Issues with Prometheus exemplars: - Prometheus still has only experimental support for exemplars after more than three years since they were introduced. It stores exemplars in memory, so they are lost after Prometheus restart. This doesn't look like production-ready feature. See `0a2f3b3794/content/docs/instrumenting/exposition_formats.md (L153-L159)` and https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage - It is very non-trivial to expose exemplars alongside metrics in your application, since the official Prometheus SDKs for metrics' exposition ( https://prometheus.io/docs/instrumenting/clientlibs/ ) either have very hard-to-use API for exposing histograms or do not have this API at all. For example, try figuring out how to expose exemplars via https://pkg.go.dev/github.com/prometheus/client_golang@v1.19.1/prometheus . - It looks like exemplars are supported for Histogram metric types only - see https://pkg.go.dev/github.com/prometheus/client_golang@v1.19.1/prometheus#Timer.ObserveDurationWithExemplar . Exemplars aren't supported for Counter, Gauge and Summary metric types. - Grafana has very poor support for Prometheus exemplars. It looks like it supports exemplars only when the query contains histogram_quantile() function. It queries exemplars via special Prometheus API - https://prometheus.io/docs/prometheus/latest/querying/api/#querying-exemplars - (which is still marked as experimental, btw.) and then displays all the returned exemplars on the graph as special dots. The issue is that this doesn't work in production in most cases when the histogram_quantile() is calculated over thousands of histogram buckets exposed by big number of application instances. Every histogram bucket may expose an exemplar on every timestamp shown on the graph. This makes the graph unusable, since it is litterally filled with thousands of exemplar dots. Neither Prometheus API nor Grafana doesn't provide the ability to filter out unneeded exemplars. - Exemplars are usually connected to traces. While traces are good for some I doubt exemplars will become production-ready in the near future because of the issues outlined above. Alternative to exemplars: Exemplars are marketed as a silver bullet for the correlation between metrics, traces and logs - just click the exemplar dot on some graph in Grafana and instantly see the corresponding trace or log entry! This doesn't work as expected in production as shown above. Are there better solutions, which work in production? Yes - just use time-based and label-based correlation between metrics, traces and logs. Assign the same `job` and `instance` labels to metrics, logs and traces, so you can quickly find the needed trace or log entry by these labes on the time range with the anomaly on metrics' graph. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5982	2024-07-03 16:09:18 +02:00
Aliaksandr Valialkin	a5d60ad78e	app/vmagent/remotewrite,lib/streamaggr: re-use common code in tests after `879771808b` - Export streamaggr.LoadFromData() function, so it could be used in tests outside the lib/streamaggr package. This allows removing a hack with creation of temporary files at TestRemoteWriteContext_TryPush_ImmutableTimeseries. - Move common code for mustParsePromMetrics() function into lib/prompbmarshal package, so it could be used in tests for building []prompbmarshal.TimeSeries from string. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6206	2024-07-03 15:22:51 +02:00
Artem Navoiev	db5b9ca8cd	github acitons: ghaction-import-gpg v5 -> v6 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-03 15:22:50 +02:00
Github Actions	bc0303bca3	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@6182902 (#6572 )	2024-07-03 15:22:50 +02:00
Github Actions	730088caaf	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@a04844a (#6571 )	2024-07-03 15:22:50 +02:00
Github Actions	a72d0310c0	Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@a53ccd7 (#6570 )	2024-07-03 14:20:56 +02:00
Aliaksandr Valialkin	4268a310c1	app/vmagent/remotewrite/remotewrite.go: make remoteWriteCtx.TryPush code easier to follow Move the code responsible for relabelCtx clearing into deferred function. This allows making more clear the remoteWriteCtx.TryPush code. This is a follow-up for `879771808b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205 While at it, clarify the description of the bugfix at docs/CHANGELOG.md	2024-07-03 14:18:51 +02:00
Aliaksandr Valialkin	f406764ccc	app/vmagent/remotewrite/streamaggr.go: clarify the description for -remoteWrite.streamAggr.* command-line flags, so they are applied to the corresponding -remoteWrite.url	2024-07-03 14:18:51 +02:00
Artem Navoiev	f327eac81b	add file placeholder grafana logs datasource docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-03 14:18:50 +02:00
Artem Navoiev	7ddaf51414	remove unsued image Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-03 14:18:50 +02:00
Github Actions	3a11c84ee8	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@dae8560 (#6568 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Artem Navoiev <tenmozes@gmail.com>	2024-07-03 14:18:49 +02:00
Aliaksandr Valialkin	f9b64cdfe0	docs/MetricsQL.md: document which metric types are usually passed to which rollup functions This should reduce incorrect usage of rollup functions like this one - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3974#issuecomment-2205574667	2024-07-03 11:36:58 +02:00
Aliaksandr Valialkin	547aa6108e	docs: update VictoriaMetrics release from v1.100.1 to v1.101.0 across all the docs This is a follow-up for `5e8c087d42` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6194	2024-07-03 01:07:07 +02:00
Aliaksandr Valialkin	bb7406e9c0	app/vmselect/promql: follow-up for `dd0d2c77c8` and `6149adbe10` Use metricsql.IsLikelyInvalid() function for determining whether the given query is likely invalid, e.g. there is high change the query is incorrectly written, so it will return unexpected results. The query is invalid most of the time if it passes something other than series selector into rollup function. For example: - rate(sum(foo)) - rate(foo + bar) - rate(foo > bar) Improtant note: the query is considered valid if it misses the lookbehind window in square brackes inside rollup function, e.g. rate(foo), since this is very convenient MetricsQL extention to PromQL, and this query returns the expected results most of the time. Other unsafe query types can be added in the future into metricsql.IsLikelyInvalid(). TODO: probably, the -search.disableImplicitConversion command-line flag must be set by default in the future releases of VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6180 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6450	2024-07-03 00:46:56 +02:00
Aliaksandr Valialkin	82748b2b9d	deployment/docker: update Go builder from Go1.22.4 to Go1.22.5 See https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved	2024-07-03 00:07:55 +02:00
Aliaksandr Valialkin	7c9212afeb	vendor: run `make vendor-update`	2024-07-03 00:00:23 +02:00
Aliaksandr Valialkin	f8779d1ed2	lib/streamaggr: follow-up for the commit `c0e4ccb7b5` - Clarify docs for `Ignore aggregation intervals on start` feature. - Make more clear the code dealing with ignoreFirstIntervals at aggregator.runFlusher() functions. It is better from readability and maintainability PoV using distinct a.flush() calls for distinct cases instead of merging them into a single a.flush() call. - Take into account the first incomplete interval when tracking the number of skipped aggregation intervals, since this behaviour is easier to understand by the end users. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6137	2024-07-02 21:34:48 +02:00
Aliaksandr Valialkin	41f95d90f9	README.md: sync with docs/Cluster-VictoriaMetrics.md after the commit `ae76794a19` This fixes `make docs-sync` results in cluster branch Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3923	2024-07-02 18:56:21 +02:00
hagen1778	e423c4b72c	docs: mention `graphite.sanitizeMetricName` in cluster docs Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ffb49c677b`)	2024-07-02 17:18:07 +02:00
Andrii Chubatiuk	252aa5a3ab	lib/protoparser/graphite: added -graphite.sanitizeMetricName flag (#6489 ) ### Describe Your Changes Added flag to sanitize graphite metrics fixes #6077 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `476faf5578`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-02 17:16:00 +02:00
Zakhar Bessarab	4ece1747d3	vmbackupmanager: fix state restore (#773 ) * app/vmbackupmanager: fix state restore on startup Fix improperly treating completed backups as "failed" in metrics. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/changelog: document the fix Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f3831bdd13`)	2024-07-02 14:37:15 +02:00
LHHDZ	c8431c8e4d	app/vmauth: reader pool to reduce gc & mem alloc (#6533 ) follow up https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6446 issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445 --------- Signed-off-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: f41gh7 <nik@victoriametrics.com> (cherry picked from commit `4d66e042e3`)	2024-07-02 14:37:15 +02:00
Github Actions	3fda4cc4de	Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@e9bd079 (#6564 ) (cherry picked from commit `dd97dd6373`)	2024-07-02 14:37:15 +02:00
hagen1778	309a767fc5	dashboards: fix wrong templating for vmauth Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e45d80cd79`)	2024-07-02 14:37:15 +02:00
Aliaksandr Valialkin	fed596b202	docs/VictoriaLogs/CHANGELOG.md: cut v0.27.0-victorialogs	2024-07-02 01:43:54 +02:00
Aliaksandr Valialkin	7426b40250	lib/logstorage: allow writing `after N` in front of `before N` at `stream_context` pipe	2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin	0912a652d5	app/vlinsert/insertutils: flush the ingested logs from in-memory buffer to storage every second Previously the in-memory buffer could remain unflushed for long periods of time under low ingestion rate. The ingested logs weren't visible for search during this time.	2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin	ab28a1f93e	app/vlinsert/syslog: add an ability to use log ingestion time as the _time field	2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin	f4dd9bd988	docs/VictoriaLogs/CHANGELOG.md: use new url https://docs.victoriametrics.com/victorialogs/querying/ instead of old one https://docs.victoriametrics.com/VictoriaLogs/querying/ (cherry picked from commit `387b3b7fb7`)	2024-07-01 16:40:44 +02:00
Hui Wang	085bc1f15c	vmui: increase max query tab from 4 to 10 (#6546 ) (cherry picked from commit `9da78f1e0e`)	2024-07-01 16:40:42 +02:00
Andrii Chubatiuk	65c742d976	deployment: remove snap packages support (#6543 ) ### Describe Your Changes Removed snap packages support as it requires time for maintenance and it's not popular at all ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `0a42c8fd8b`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-01 16:40:39 +02:00
Hui Wang	87cb132f53	app/vmselect/netstorage: do not retry request when complexity limit i… (#6469 ) …s already exceeded --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-07-01 16:38:15 +02:00
Andrii Chubatiuk	937ae2ca90	lib/streamaggr: added stale samples metric, added metrics labels (#6462 ) ### Describe Your Changes - added stale metrics counters for input and output samples - added labels for aggregator metrics => `name="{rwctx}:{aggrId}:{aggrSuffix}"` - rwctx - global or number starting from 1 - aggrid - aggregator id starting from 1 - aggrSuffix - <interval>_(by\|without)_label1_label2_labeln e.g: `name="global:1:1m_without_instance_pod"` ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `861852f262`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-07-01 15:01:49 +02:00
hagen1778	69625aa8a1	docs: mark `without` optional in stream aggr docs Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f8eea0f2c9`)	2024-07-01 14:56:46 +02:00
Aliaksandr Valialkin	5143e93534	docs/VictoriaLogs/CHANGELOG.md: cut v0.26.1-victorialogs	2024-07-01 02:33:02 +02:00
Aliaksandr Valialkin	208a624d4d	lib/logstorage: properly search for the surrounding logs in `stream_context` pipe The set of log fields in the found logs may differ from the set of log fields present in the log stream. So compare only the log fields in the found logs when searching for the matching log entry in the log stream. While at it, return _stream field in the delimiter log entry, since this field is used by VictoriaLogs Web UI for grouping logs by log streams.	2024-07-01 02:33:00 +02:00
Aliaksandr Valialkin	516c7b2ca0	docs/VictoriaLogs: remove "preview" warning - VictoriaLogs is ready for prod	2024-07-01 01:52:40 +02:00
Aliaksandr Valialkin	0e6ee55a95	docs/VictoriaLogs/CHANGELOG.md: cut v0.26.0-victorialogs	2024-07-01 01:49:55 +02:00
Aliaksandr Valialkin	76a58ae08d	lib/logstorage: add ability to store sorted log position into a separate field with `sort ... rank <fieldName>` syntax	2024-07-01 01:46:03 +02:00
Aliaksandr Valialkin	d0dca7b8c5	lib/logstorage: add delimiter between log chunks returned from `\| stream_context` pipe	2024-07-01 01:46:02 +02:00
Artem Navoiev	2ac2d0919d	operator: sync docs with operator repo, add alias Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-06-29 17:19:49 +02:00
Aliaksandr Valialkin	76053a0ef0	docs/VictoriaLogs: typo fixes	2024-06-28 19:26:32 +02:00
Aliaksandr Valialkin	75c7b2c07a	docs/VictoriaLogs/CHANGELOG.md: cut v0.25.0-victorialogs	2024-06-28 19:18:21 +02:00
Aliaksandr Valialkin	4b3477e62b	lib/logstorage: add `stream_context` pipe, which allows selecting surrounding logs for the matching logs	2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin	c9fc8079c4	app/vlinsert/syslog: properly skip empty lines in Syslog protocol Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548	2024-06-28 14:09:45 +02:00
Aliaksandr Valialkin	bb6424aeca	app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545 in order to return top N groups with the biggest number of hits.	2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin	3eecc3de8c	docs/VictoriaLogs: typo fixes	2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin	5a24ea6cc3	docs/VictoriaLogs: document how to export logs from VictoriaLogs While at it, mention vmctl-like tool for migrating logs from other systems to VictoriaLogs at docs/VictoriaLogs/Roadmap.md Thanks to the question from Xavier Pestel at https://www.linkedin.com/feed/update/urn:li:activity:7212093021301927937?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7212093021301927937%2C7212171550651731969%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287212171550651731969%2Curn%3Ali%3Aactivity%3A7212093021301927937%29	2024-06-28 02:06:24 +02:00
Aliaksandr Valialkin	2f28819bb1	lib/logstorage: it is safe using `\| unroll` pipe in live tailing `\| unroll` pipe can make multiple copies of rows from the input row. This doesn't break live tailing, so allow `\| unroll` pipe in live tailing.	2024-06-27 19:45:12 +02:00
Aliaksandr Valialkin	05365999ea	docs/VictoriaLogs/querying/README.md: clarify live tailing docs	2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin	4391ef25e9	docs/VictoriaLogs/Roadmap.md: remove `Web UI` from roadmap, since it is already implemented See https://docs.victoriametrics.com/victorialogs/querying/#web-ui While at it, add DataDog protocol for logs into 'data ingestion protocols' roadmap.	2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin	00899506f7	docs/VictoriaLogs/querying: remove unused and outdated vmui.webp image The image has been deleted from the docs/VictoriaLogs/querying/README.md file in the commit `87f1c8bd6c`	2024-06-27 15:16:43 +02:00
Aliaksandr Valialkin	13df75a28b	docs/VictoriaLogs/querying/README.md: add a link to live tailing docs from `querying log` docs	2024-06-27 15:12:59 +02:00
Aliaksandr Valialkin	b26acec9a8	app/vlselect: properly return live tailing results	2024-06-27 15:06:15 +02:00
Aliaksandr Valialkin	dd62a2b9d6	lib/logstorage: work-in-progress	2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin	0352d98d74	docs/goals.md: clarify that the usability of the existing functionality must be improved the first	2024-06-27 09:55:40 +02:00
Artem Navoiev	19c4dfd72c	dashboards: update statistic by tenant dashboard, fix billing disk usage pie panel (#6521 ) - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c9f496bdd0`)	2024-06-27 09:32:13 +02:00
Arkadii Yakovets	c4bae2ab77	github/workflows: run CodeQL analysis for both JavaScript and TypeScript (#6541 ) ### Describe Your Changes Enables CodeQL TypeScript analysis. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> (cherry picked from commit `8a982e88cd`)	2024-06-27 09:32:11 +02:00
Arkadii Yakovets	5a199d7e8f	github/workflows: run CodeQL Go analysis for .go files only (#6540 ) ### Describe Your Changes The current CodeQL setup runs Go analysis on irrelevant files (see [this run](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/runs/9380962792) as an example that took around 3 minutes, [PR link](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419)). The PR makes sure the Go CodeQL job is triggered by `.go` path pattern only. It contains 2 commits -- first for a better diff, and second for renaming the config file. ### Checklist The following checks are mandatory**: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> (cherry picked from commit `c3f8af5636`)	2024-06-27 09:32:09 +02:00
Josh Keife	bdebc128f7	Update vmalert.md (#6539 ) Fix typo ### Describe Your Changes Fix typo in the vmalert docs. In the docs it states rules at `VMAgent`'s namespace when it should be `VMAlert`'s namespace. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `8d01dd0514`)	2024-06-27 09:32:07 +02:00
Aliaksandr Valialkin	911891da95	docs/goals.md: clarify that the listed goals are related to development and support, not sales and marketing	2024-06-26 19:12:36 +02:00
Fred Navruzov	ed41f18c5a	docs/vmanomaly: quickstart page improvements (#6538 ) ### Describe Your Changes Small improvements to a QuickStart guide of `vmanomaly` ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-26 19:12:35 +02:00
Andrii Chubatiuk	580d02c3f8	added IMDSv2 for YC SD (#6524 ) ### Describe Your Changes Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-26 19:12:35 +02:00
Andrii Chubatiuk	f79df2aa8b	app/vmauth: allow dropping host header (#6525 ) ### Describe Your Changes Fixes #6453 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-26 19:12:35 +02:00
rtm0	48a5c4cb01	Fix Date metricid cache consistency under concurrent use (#6534 ) ### Describe Your Changes Fix Date metricid cache consistency under concurrent use. When one goroutine calls Has() and does not find the cache entry in the immutable map it will acquire a lock and check the mutable map. And it is possible that before that lock is acquired, the entry is moved from the mutable map to the immutable map by another goroutine causing a cache miss. The fix is to check the immutable map again once the lock is acquired. ### Checklist The following checks are mandatory: - [x ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-06-26 19:12:34 +02:00
Aliaksandr Valialkin	3a2c0b0a61	docs/goals.md: heading fixes	2024-06-26 16:41:21 +02:00
Aliaksandr Valialkin	d2be74b4a1	docs/goals.md: add VictoriaMetrics proverbs	2024-06-26 14:01:15 +02:00
Aliaksandr Valialkin	75d45068a5	docs/VictoriaLogs/README.md: document the relation between `-retention.maxDiskSpaceUsageBytes` and `-retentionPeriod` command-line flags	2024-06-26 12:03:29 +02:00
Aliaksandr Valialkin	db5a5a4f6e	docs/goals.md: add additional non-goals, which frequently become false goals according to the development history of VictoriaMetrics	2024-06-26 12:03:28 +02:00
Yury Molodov	6bde0196d8	vmui/logs: fix the update of the relative time range (#6517 ) ### Describe Your Changes - Fixed the update of the relative time range when `Execute Query` is clicked - Optimized server requests: now, if an error occurs in the `/query` request, the `/hits` request will not be executed. #6345 (duplicates: #6440, #6312) (cherry picked from commit `43342745ac`)	2024-06-26 11:26:08 +02:00
James Rhoat	8249de986e	Updating documentation around OTEL (#6519 ) Updating documentation around the opentelemetry endpoint for metrics and the "How to use OpenTelemetry metrics with VictoriaMetrics" guide so that it shows not only how to directly write but also how to write to the otel collector and view metrics in vmui. --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `6652fb630f`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-26 11:26:05 +02:00
Yury Molodov	904ec020ed	vmui: fix input cursor position reset (#6530 ) ### Describe Your Changes This PR addresses the issue where the cursor jumps to the end of the input fields in the modal settings window after each keystroke. ### Before fix: ![ezgif-7-4c69805cea](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/2e99e833-09e3-4b44-89aa-fc1bd3c4346d) ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `e9b71a2883`)	2024-06-26 11:25:47 +02:00
Yury Molodov	25f3e700a6	vmui: update package-lock.json (#6532 ) 1. Updated `package-lock.json` to resolve [Dependabot alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/security/dependabot). 2. Updated types to align with the latest `Preact` update. (cherry picked from commit `6cab811134`)	2024-06-26 11:25:45 +02:00
hagen1778	6ad66b3317	docs: rm mentions of `vmrecover` as it still doesn't exist While there, add a hint about truncated error messages. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6775a00622`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-26 11:25:42 +02:00
Daria Karavaieva	affde1794f	docs/vmanomaly:quickstart example fix (#6535 ) - Fixed config example in QuickStart vmanomaly docs for 1.13 version compatibility - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `4b77778ff5`)	2024-06-26 11:25:15 +02:00
Aliaksandr Valialkin	3c59e424c4	Makefile: add `clean-checkers` rule for dropping checker apps used in `make check-all` The `make clean-checkers` command may be useful when the locally installed checker apps are too old, so they must be updated to the newly requested versions by `make check-all`. In this case the fix is to run make clean-checkers check-all	2024-06-25 20:30:25 +02:00
Aliaksandr Valialkin	2dfbc40563	docs/goals.md: add VictoriaMetrics goals and non-goals page Everybody who works on VictoriaMetrics must be aware of these goals and non-goals	2024-06-25 18:00:55 +02:00
Aliaksandr Valialkin	d5cbda3424	app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage	2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin	252ff71a49	docs/VictoriaLogs/data-ingestion/README.md: document that Rsyslog and Syslog-ng can be used for ingesting logs into VictoriaLogs	2024-06-25 15:15:53 +02:00
Aliaksandr Valialkin	0a88a55ae0	docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release	2024-06-25 14:59:31 +02:00
Aliaksandr Valialkin	f24123a776	lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data	2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin	1716c4e609	lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets. While at it, make tryParseTimestampISO8601 function private in order to prevent from improper usage of this function from outside the lib/logstorage package. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508	2024-06-25 14:54:24 +02:00
hagen1778	26d58f8768	docs: add missing ticket links to the recent releases Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-25 14:54:24 +02:00
hagen1778	aa474c06b9	deployment: set consistent Grafana version for VM datasource env Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `bb0449d7ba`)	2024-06-25 13:59:00 +02:00
hagen1778	841305da8a	deployment: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `0dcb3acbf7`)	2024-06-25 13:58:59 +02:00
hagen1778	8ab51ac2b1	deployment: simplify Grafana plugins installation rm custom scripts for downloading Grafana plugins for VictoriaMetrics and VictoriaLogs. Use `GF_INSTALL_PLUGINS` instead. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5f218a772b`)	2024-06-25 13:58:59 +02:00
Nikolay	bf1464fc33	dashboards: add dashboard and alerts for vmauth (#6491 ) Signed-off-by: f41gh7 <nik@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `14b9ef1e4d`)	2024-06-25 11:17:04 +02:00
hagen1778	8237b63fe3	docs: make docs-sync Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `cc6a1da0d7`)	2024-06-25 11:17:04 +02:00
Roman Khavronenko	e3da10a7c1	docs: add section for docs modification (#6528 ) While there, remove the obsolete files from the previous version of docs. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `87edfa61c1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-25 11:17:03 +02:00
Aliaksandr Valialkin	30d1f0711f	app/vmselect/netstorage: add a comment explaining why all the samples in block are taken into account when checking the -search.maxSamplesPerQuery limit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851 This is a follow-up for `b07a02c516`	2024-06-25 03:06:42 +02:00
Aliaksandr Valialkin	2a7fcba330	lib/logstorage: make golangci-lint happy	2024-06-25 03:06:28 +02:00
Aliaksandr Valialkin	a5445e09c2	Revert "app/vmselect: fix the way of counting raw samples in single query (#6464 )" This reverts commit `5ecf439078`. Reason for revert: the previous logic was correct. The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources, which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits . VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it, then filters out samples outside the selected time range. This means that it _spends CPU time_ on reading and unpacking of _all the samples_ in every block on the requested time range, even if only a single sample per each block matches the given time range. The previous logic was effectively limiting CPU time a single query could take. The new logic fails limiting CPU time a single query could take in some pathological cases when only a small fraction of samples per each requested block fit the requested time range. This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks, which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion, and the query requests a single sample out of 8K samples per each block, this means that the query may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples. This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464	2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin	813a7aec6c	docs: return back spellcheck fixes from the commit `92b22581e6`, which were accidentally reverted in `45cf83cd8f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470	2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin	7f2b491748	Revert "docs: Add spellcheck workflow (#6470 )" This reverts commit `92b22581e6`. Reason for revert: too complex and slow approach for spellchecking task. This approach may significantly slow down development pace. It also may take non-trivial amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory. Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed (e.g. once per month or once per quarter).	2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin	7026498359	lib/httpserver: revert `9b7e532172` Reason for revert: this commit doesn't resolve real security issues, while it complicates the resulting code in subtle ways (aka security circus). Comparison of two strings (passwords, auth keys) takes a few nanoseconds. This comparison is performed in non-trivial http handler, which takes thousands of nanoseconds, and the request handler timing is non-deterministic because of Go runtime, Go GC and other concurrently executed goroutines. The request handler timing is even more non-deterministic when the application is executed in shared environments such as Kubernetes, where many other applications may run on the same host and use shared resources of this host (CPU, RAM bandwidth, network bandwidth). Additionally, it is expected that the passwords and auth keys are passed via TLS-encrypted connections. Establishing TLS connections takes additional non-trivial time (millions of nanoseconds), which depends on many factors such as network latency, network congestion, etc. This makes impossible to conduct timing attack on passwords and auth keys in VictoriaMetrics components. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6423/files Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392	2024-06-25 01:51:06 +02:00
Aliaksandr Valialkin	7de6f5b4ce	lib/logstorage: work-in-progress	2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin	8dd69fd7ae	docs/CHANGELOG.md: cut v1.102.0-rc2 release	2024-06-24 23:22:45 +02:00
Aliaksandr Valialkin	f8ff09cd8d	app/vmui: run `make vmui-update` after `65f414acee`	2024-06-24 23:21:19 +02:00
Aliaksandr Valialkin	faed1394d9	app/vmctl/prometheus/prometheus.go: add missing arg to tsdb.OpenDBReadOnly() function after updating github.com/prometheus/prometheus dependency from v0.52.1 to v0.53.0 in `5c55722db4` See `c5a1cc9148`	2024-06-24 23:16:30 +02:00
Aliaksandr Valialkin	14d52a749b	vendor: run `make vendor-update`	2024-06-24 23:09:16 +02:00
Andrii Chubatiuk	516848783e	deployment: build image for vmagent streamaggr benchmark (#6515 ) ### Describe Your Changes optionally build vmagent image for benchmark needed for https://github.com/VictoriaMetrics/ops/pull/1297 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `6b128da811`)	2024-06-24 16:29:14 +02:00
Nikolay	7ab046cf88	vendor update: updates metrics to v1.34.0 (#6523 ) mention change for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457 ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `1af13208c2`)	2024-06-24 14:52:32 +02:00
Ivan Yatskevich	50343768fd	Publish Alertmanager configuration docs for Managed VictoriaMetrics (#6428 ) ### Describe Your Changes With the recent release of Managed VictoriaMetrics users are able to create and execute Alerting & Recording rules and send notifications via hosted Alertmanager. So, we're publishing Alertmanager configuration docs for Managed VictoriaMetrics. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `1106d91741`)	2024-06-24 12:50:15 +02:00
Arkadii Yakovets	c454084eb1	docs: Add spellcheck workflow (#6470 ) ### Describe Your Changes This is a follow-up PR containing workflow related part of the initial [PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362). It adds spell checking control based on [cspell](https://github.com/streetsidesoftware/cspell). The related configs are consolidated under `cspell` directory. You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt` (it contains ~30 items atm). All other absent commonly used terms should be added directly to respective [cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md). ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> (cherry picked from commit `92b22581e6`)	2024-06-24 12:50:13 +02:00
hagen1778	63c15d76cd	dashboards: fix typo in panel descriptions for vmagent Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b201d1722d`)	2024-06-21 11:44:29 +02:00
hagen1778	1aadf2b267	app/vmalert: fix typo in replay error handling Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `279815818c`)	2024-06-20 15:15:59 +02:00
hagen1778	d98577ae37	app/vmalert: follow-up `bc37b279aa` * rm extra interface method for rw Client, as it has low applicability and doesn't fit multitenancy well * add `GetDroppedRows` method instead Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4ef76eed7b`)	2024-06-20 15:15:58 +02:00
Andrii Chubatiuk	50783fca4d	app/vmagent: add max_scrape_size to scrape config (#6434 ) Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1e83598be3`)	2024-06-20 14:00:22 +02:00
Hui Wang	a393b993d6	vmalert: exit replay mode with non-zero code if generated samples are… (#6513 ) … not successfully written into remoteWrite url address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512 (cherry picked from commit `bc37b279aa`)	2024-06-20 14:00:19 +02:00
Slava Bobik	a7266785ce	Fixed a typo in the FastQueue mutex comment (#6514 ) ### Describe Your Changes Fixed a small typo in a comment about the mutex inside the FastQueue struct ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `d236604d39`)	2024-06-20 14:00:08 +02:00
Aliaksandr Valialkin	d5224f3363	lib/logstorage: work-in-progress	2024-06-20 03:10:37 +02:00
Zakhar Bessarab	886f545f81	lib/fs/fscore: do not trim content from path (#6503 ) ### Describe Your Changes Trimming content which is loaded from an external pass leads to obscure issues in case user-defined input contained trimmed chars. For example. user-defined password "foo\n" will become "foo" while user will expect it to contain a new line. --- For example, a user defines a password which ends with `\n`. This often happens when user Kubernetes secrets and manually encodes value as base64-encoded string. In this case vmauth configuration might look like: ``` users: - url_prefix: - http://vminsert:8480/insert/0/prometheus/api/v1/write name: foo username: foo password: "foobar\n" ``` vmagent configuration for this setup will use the following flags: ``` -remoteWrite.url=http://vmauth:8427/ -remoteWrite.basicAuth.passwordFile=/tmp/vmagent-password -remoteWrite.basicAuth.username="foo" ``` Where `/tmp/vmagent-password` is a file with `foobar\n` password. Before this change such configuration will result in `401 Unauthorized` response received by vmagent since after file content will become `foobar`. --- An example with Kubernetes operator which uses a secret to reference the same password in multiple configurations. <details> <summary>See full manifests</summary> `Secret`: ``` apiVersion: v1 data: name: Zm9v # foo password: Zm9vYmFy # foobar\n username: Zm9v= # foo kind: Secret metadata: name: vmuser ``` `VMUser`: ``` apiVersion: operator.victoriametrics.com/v1beta1 kind: VMUser metadata: name: vmagents spec: generatePassword: false name: vmagents targetRefs: - crd: kind: VMAgent name: some-other-agent namespace: example username: foo # note - the secret above is referenced to provide password passwordRef: name: vmagent key: password ``` `VMAgent`: ``` apiVersion: operator.victoriametrics.com/v1beta1 kind: VMAgent metadata: name: example spec: selectAllByDefault: true scrapeInterval: 5s replicaCount: 1 remoteWrite: - url: "http://vmauth-vmauth-example:8427/api/v1/write" # note - the secret above is referenced as well basicAuth: username: name: vmagent key: username password: name: vmagent key: password ``` </details> Since both config target exactly the same `Secret` object it is expected to work, but apparently the result will be `401 Unauthrized` error. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `201fd6de1e`)	2024-06-19 10:37:12 +02:00
Hui Wang	7a21e6cb6b	vmalert-dashboard: replace variable query metric (#6505 ) `vmalert_iteration_total` series number is 4 time less than `vmalert_iteration_duration_seconds`, queries will be lighter. (cherry picked from commit `75ad6c1b49`)	2024-06-19 10:37:10 +02:00
Nihal	8fd46caa22	victoria-metrics: constant-time comparison of credentials like authkeys and basic auth credentials (#6423 ) Changes for constant-time comparison of credentials like authkeys and basic auth credentials. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392 --------- Signed-off-by: Syed Nihal <syed.nihal@nokia.com> (cherry picked from commit `9b7e532172`)	2024-06-19 10:37:09 +02:00
Yury Molodov	5e8e89f22d	vmui/logs: update footer links (#6498 ) ### Describe Your Changes Update the links in the footer for logs: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) and [Documentation](https://docs.victoriametrics.com/victorialogs/) (cherry picked from commit `13e3bb88a9`)	2024-06-18 15:29:14 +02:00
Hui Wang	36f6a08f07	doc: add missing cmd flag description (#6504 ) (cherry picked from commit `2259e3cac7`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-18 15:29:11 +02:00
Yury Molodov	88650abf97	vmui/logs: add bar chart (#6461 ) - Added a bar chart displaying the number of log entries over a time range. #6404 - When `_msg` is empty, all fields are displayed in a single line. - Added double quotes when copying pairs: `key: "value"`. - Minor style adjustments. (cherry picked from commit `32fbffedd9`)	2024-06-18 15:28:56 +02:00
Hui Wang	5be2f2c4e4	vmalert-tool: support file path with hierarchical patterns and regexp… (#6501 ) …es, and http url in unittest cmd-line flag `-files` (cherry picked from commit `3b8970802e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-18 14:17:42 +02:00
Hui Wang	b09da16899	vmalert-tool: exit normally when no rule is defined under rule group (#6502 ) address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6500 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d62f303e53`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-18 14:17:23 +02:00
Aliaksandr Valialkin	8650809435	docs/keyConcepts.md: document optional `timeout` query arg at /api/v1/query and /api/v1/query_range This is a follow-up for `ac9c2a796f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6020	2024-06-18 12:32:04 +02:00
Aliaksandr Valialkin	0ee360b48c	docs/VictoriaLogs/CHANGELOG.md: typo fix in the release date for v0.20.2-victorialogs	2024-06-18 00:28:49 +02:00
Aliaksandr Valialkin	ee114ca59c	app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint This has been broken in `2b6a634ec0`	2024-06-18 00:24:11 +02:00
Aliaksandr Valialkin	7658cc36a6	docs/VictoriaLogs/CHANGELOG.md: cut v0.20.1-victorialogs	2024-06-17 23:29:28 +02:00
Aliaksandr Valialkin	c10a646d19	app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports	2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin	f74e6b0674	app/vlinsert: properly parse length-delimited syslog messages sent over TCP according to RFC5425	2024-06-17 22:30:28 +02:00
jackyin	4a6bf7f218	app/vmui: copy button shows undefined (#6495 ) ### Describe Your Changes fix #6421 some aggregation func don't return \_\_name\_\_ value	2024-06-17 22:30:28 +02:00
hagen1778	863f1c2513	lib/streamaggr: remove accidentally committed changes Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `34771ab293`)	2024-06-17 14:25:45 +02:00
Roman Khavronenko	df7e300071	app/vmselect/promql: check for ranged vectors in aggr funcs if implicit conversions are disabled (#6450 ) Check for ranged vector arguments in aggregate expressions when `-search.disableImplicitConversion` or `-search.logImplicitConversion` are enabled. For example, `sum(up[5m])` will fail to execute if these flags are set. ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [*] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6149adbe10`)	2024-06-17 14:25:43 +02:00
Aliaksandr Valialkin	938862946c	docs/VictoriaLogs: mention which collectors support sending logs to VictoriaLogs in syslog format	2024-06-17 13:13:51 +02:00
Aliaksandr Valialkin	1750991119	lib/logstorage: work-in-progress	2024-06-17 12:13:25 +02:00
Zhu Jiekun	9c3dabccac	docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494 ) ### Describe Your Changes Add CHANGELOG for Statsd support in v1.102.0-rc1 - CHANGELOG is missing for #5053. - It would be better to include it so that users can be aware of when this feature is released, rather than attempting to use it with lower version VM components. See discussion [here](https://victoriametrics.slack.com/archives/CGZF1H6L9/p1718462859065049?thread_ts=1718451117.668789&cid=CGZF1H6L9). ### Checklist The following checks are mandatory: - [X] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-17 12:11:50 +02:00
Fred Navruzov	73fbce30bd	docs/vmanomaly - add config example for min_dev arg (#6490 ) ### Describe Your Changes Added config example for `min_dev_from_expected` arg; also, small styling fixes and alignments ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-17 12:11:50 +02:00
hagen1778	43c53271bf	docs: update wording for `6e395048d3` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `da4fbf61a4`)	2024-06-14 16:31:42 +02:00
Hui Wang	5ecf439078	app/vmselect: fix the way of counting raw samples in single query (#6464 ) The limit is specified with command-line flag `-search.maxSamplesPerQuery`. Previously, samples might be over-counted and query can't be fixed by reducing time range. address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851 (cherry picked from commit `6e395048d3`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-14 16:24:31 +02:00
Andrii Chubatiuk	8ca1813bd2	lib/flagutil: use month limit for duration flag for parsed duration assessment (#6486 ) use maxMonths limit for parsed duration flag value https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `faf67aa8b5`)	2024-06-14 15:21:32 +02:00
jackyin	f69495cd5f	app/vmalert: fix VMAlert oauth2 error (#6478 ) Properly set ClientSecret param for notifier. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471 --------- Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5223981fed`)	2024-06-14 15:21:30 +02:00
hagen1778	64df1383e7	app/vmalert: properly configure authentication with S3 when `-s3.configFilePath` is specified. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-14 14:22:35 +02:00
Andrii Chubatiuk	abc233a902	lib/backup/s3remote: fixed credsFilePath flag (#6488 ) properly use credsFilePath flag value https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e678a9aa51`)	2024-06-14 14:14:58 +02:00
Zakhar Bessarab	a91a2b97ed	docs/guides/otel-guide: update sentence to make it less obscure (#6483 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `f11261dc44`)	2024-06-14 14:14:56 +02:00
Andrii Chubatiuk	779436bd9c	app/vmalert: fixed path prefixes for system routes (#6435 ) Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `eea361defb`)	2024-06-14 14:14:54 +02:00
Artem Navoiev	6c587eefbf	docs: add more redirects Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `d1baa41b6a`)	2024-06-14 13:26:47 +02:00
Fred Navruzov	455d761de9	docs/vmanomaly: fix-typos-in-v1.13-preset (#6484 ) ### Describe Your Changes small fix of typos in v1.13 presets (vmanomaly docs) ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `1cf28598c6`)	2024-06-14 13:26:44 +02:00
Roman Khavronenko	5df50e5645	lib/streamaggr: prevent `rate_sum` and `rate_avg` from producing NaNs (#6482 ) ### Describe Your Changes * check if `lastValue` was seen at least twice with different timestamps. Otherwise, the difference between last timestamp and previous timestamp could be `0` and will result into `NaN` calculation * check if there items left in lastValue map after staleness cleanup. Otherwise, `rate_avg` could have produce `NaN` result. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `51d19485bb`)	2024-06-14 13:26:42 +02:00
Github Actions	5683610bf8	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@bc562ef (#6481 ) (cherry picked from commit `2d575cd27a`)	2024-06-14 13:26:40 +02:00
Fred Navruzov	aa85b772bb	docs/vmanomaly: update preset guide and cross-links (#6480 ) ### Describe Your Changes Updates and cross-references for `preset` mode of `vmanomaly` [in docs](https://docs.victoriametrics.com/anomaly-detection/presets/) ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `2184c0c730`)	2024-06-14 13:26:37 +02:00
Daria Karavaieva	ae39b02496	docs/vmanomaly: preset guide (#6151 ) - added instructions of how to run presets - description of Node Exporter indicators covered (cherry picked from commit `c32b8d39dd`)	2024-06-14 13:26:29 +02:00
Aliaksandr Valialkin	b455d356fc	docs/CONTRIBUTING.md: add checks about backwards compatibility and historical conventions	2024-06-13 17:08:54 +02:00
Aliaksandr Valialkin	2bbf62b6f6	lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes Previously byte slices up to 2^20 bytes (e.g. 1Mb) were cached because of a typo in the commit `c14dafce43` . This could result in increased memory usage when vmagent scrapes many regular targets, which expose relatively small number of metrics (e.g. up to a few thousand per target) and a few large targets such as kube-state-metrics, which expose more than 10 thousand metrics. This is common case for Kubernetes monitoring. While at it, remove pools for very small byte slices, since they are rarely used during scraping.	2024-06-13 17:02:05 +02:00
Aliaksandr Valialkin	7af2b7905f	docs/CONTRIBUTING.md: various typo fixes	2024-06-13 17:02:05 +02:00
Aliaksandr Valialkin	faf07fbc67	lib/bytesutil: optimize internStringMap cleanup - Make it in a separate goroutine, so it doesn't slow down regular intern() calls. - Do not lock internStringMap.mutableLock during the cleanup routine, since now it is called from a single goroutine and reads only the readonly part of the internStringMap. This should prevent from locking regular intern() calls for new strings during cleanups. - Add jitter to the cleanup interval in order to prevent from synchornous increase in resource usage during cleanups. - Run the cleanup twice per -internStringCacheExpireDuration . This should save 30% CPU time spent on cleanup comparing to the previous code, which was running the cleanup 3 times per -internStringCacheExpireDuration .	2024-06-13 15:09:42 +02:00
Zakhar Bessarab	ac16d1dc1b	lib/promscrape: increase default value for promscrape.maxDroppedTargets to 10_000 (#6459 ) ### Describe Your Changes This limit can be increased since after `4513893ead` tracking of dropped targets uses much less memory per entry. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381#issuecomment-2156708228 ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `34071ac660`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-13 09:28:16 +02:00
LHHDZ	41e4135371	app/vmauth: fix discovering backend IPs when `url_prefix` contains hostname with `srv+` prefix (#6401 ) This change fixes the following panic: ``` 2024-06-04T11:16:52.899Z warn app/vmauth/auth_config.go:353 cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally panic: runtime error: integer divide by zero goroutine 9 [running]: github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1() /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58 panic({0x103115100?, 0x10338d700?}) /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124 main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?) /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210 main.(*URLPrefix).getBackendURL(0x140000aa080) /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8 ``` --------- Co-authored-by: Haley Wang <haley@victoriametrics.com>	2024-06-12 11:47:44 +02:00
Aliaksandr Valialkin	27efb9499b	docs/VictoriaLogs/logsql-examples.md: added an example on how to search for a phrase with whitespace	2024-06-11 19:13:47 +02:00
Aliaksandr Valialkin	9135b404d9	lib/logstorage: work-in-progress	2024-06-11 17:51:01 +02:00
Aliaksandr Valialkin	9bd16790c0	lib/streamaggr: prevent from data race inside dedupAggrShard when samplesBuf can be updated in pushSamples() while their values are read in the flush() loop without das.mu lock This issue has been introduced in the commit `253c0cffbe`	2024-06-11 17:31:38 +02:00
Aliaksandr Valialkin	81a3387bb7	.golangci.yml: properly specify functions to exclude for return values check after the upgrade to v1.59.1 at `239a7b6e6f` (cherry picked from commit `4ad1f8ee3a`)	2024-06-11 17:05:17 +02:00
Aliaksandr Valialkin	b74dbdbc82	Makefile: update golangci-lint from v1.57.1 to v1.59.1 See https://github.com/golangci/golangci-lint/releases/tag/v1.59.1 Thanks to ccoVeille for the suggestion at https://www.reddit.com/r/golang/comments/1dcl4ne/log_viewer_in_go/l82vsfd/ (cherry picked from commit `239a7b6e6f`)	2024-06-11 17:05:12 +02:00
Fred Navruzov	ea90d91a30	docs/vmanomaly - fix 404 img links & remove outdated models (#6460 ) ### Describe Your Changes Fix 404 relative img links for v1.13.0 update of vmanomaly docs ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `9a355c3512`)	2024-06-11 17:05:10 +02:00
Fred Navruzov	0219d34b21	docs/vmanomaly - release 1.13.0 preparation (#6436 ) ### Describe Your Changes [vmanomaly docs](https://docs.victoriametrics.com/anomaly-detection/) update for changes, introduced in v1.13.0 ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `1feb5d04d7`)	2024-06-11 17:05:07 +02:00
Nikolay	ae76794a19	follow-up docs update after `77f22fdb8d` (#6454 ) Signed-off-by: f41gh7 <nik@victoriametrics.com> (cherry picked from commit `33d07e915f`)	2024-06-11 17:05:05 +02:00
James Rhoat	f4b52b8137	updating operator dashboard chart to be titled working instead of wokring (#6455 ) ### Describe Your Changes Corrected spelling mistake in the operator json to be "working" instead of "wokring" ### Checklist The following checks are mandatory: - [ x ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `fbd4b8e1ab`)	2024-06-11 17:05:03 +02:00
Github Actions	c96b334b3e	Automatic update operator docs from VictoriaMetrics/operator@310a860 (#6449 ) (cherry picked from commit `020d3f3d88`)	2024-06-11 17:05:01 +02:00
Lapo Luchini	dd311aa76f	Small fix in MetricsQL.md (#6452 ) ### Describe Your Changes value→valid ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Lapo Luchini <lapo@lapo.it> (cherry picked from commit `14f586238f`)	2024-06-11 17:04:59 +02:00
Nikolay	66fbea70a5	follow-up after `77f22fdb8d` (#6458 ) * fixes linter error * simplify code a bit * fixes bug with incorrectly set configSuccess metric. It was not set to 1 in case of config rollback Signed-off-by: f41gh7 <nik@victoriametrics.com>	2024-06-11 12:08:00 +02:00
noodles2hg	77f22fdb8d	[cluster/vminsert]:add reload -relabelConfig on the request to /-/reload (#3923 ) When I use vminsert's `relabelConfig`, I found that now there is no reloaded api. However, `vminsert` under `VM-Single` has it. So, I hope to add it to the `cluster/vminster`. --------- Signed-off-by: z-anshun <1179798460@qq.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-06-10 19:36:41 +02:00
Aliaksandr Valialkin	37a8cc0b12	lib/logstorage: work-in-progress	2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin	7e24bf99de	lib/streamaggr: return back string interning to dedupAggr after 78953723200f15ffc417064d1912bdbb7551505c It should reduce memory allocation rate during stream deduplication	2024-06-10 18:06:25 +02:00
Aliaksandr Valialkin	6470eac7dc	lib/bytesutil: reduce the number of memory allocations per each interned string in bytesutil.InternString() from 5 to 1 This should reduce GC overhead when tens of millions of strings are interned (for example, during stream deduplication of millions of active time series).	2024-06-10 18:06:24 +02:00
Roman Khavronenko	8c8d84e30a	lib/protoparser/opentelemetry/firehose: escape requestID before returning it to user (#6451 ) All user input should be sanitized before rendering. This should prevent possible attacks. See https://github.com/VictoriaMetrics/VictoriaMetrics/security/code-scanning/203 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-10 18:06:24 +02:00
Aliaksandr Valialkin	883c0e6221	lib/streamaggr: reduce memory allocations by using dedupAggrSample buffer per each dedupAggrShard	2024-06-10 16:39:26 +02:00
Aliaksandr Valialkin	422225bfa5	lib/streamaggr: reduce the number of duplicates per each sample in BenchmarkDedupAggr from 100 to 2 This is closer to typical production setups when deduplication is used for de-duplicating of 2 samples per series.	2024-06-10 16:39:26 +02:00
Yury Molodov	2300e30ff3	vmui/logs: add markdown support (#6292 ) Add support for markdown format and emoji for the `_msg` field in the "Group" view. Add markdown rendering toggle. Disabled by default. Value is stored in `localStorage`.	2024-06-10 16:39:25 +02:00
Aliaksandr Valialkin	d269a95da3	lib/streamaggr: use strings.Clone() instead of bytesutil.InternString() for creating series key in dedupAggr Our internal testing shows that this reduces GC overhead when deduplicating tens of millions of active series.	2024-06-10 16:08:47 +02:00
Aliaksandr Valialkin	9ed9e766e8	lib/streamaggr: improve performance for dedupAggr.sizeBytes() and dedupAggr.itemsCount() These functions are called every time `/metrics` page is scraped, so it would be great if they could be sped up for the cases when dedupAggr tracks tens of millions of active time series.	2024-06-10 16:00:05 +02:00
Aliaksandr Valialkin	387c22da49	lib/streamaggr: remove flushState arg at dedupAggr.flush(), since it is always set to true in production	2024-06-10 16:00:05 +02:00
hagen1778	fdf0a936f0	vmctl: rm `--vm-disable-progress-bar` flag It is better to remove deprecated flag completely, so vmctl will fail if this flag is used and user can immediately fix the issue. Before, flag was ignored and it is worse then fail fast. follow-up after `8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8d95522529`)	2024-06-10 14:05:58 +02:00
Nikolay	0ce7f38e1c	app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388 ) * adds idleConnTimeout flag, which must reduce probability of `broken pipe` and `connection reset` errors. * one-time retry trivial network requests for the same backend --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d44058bcd6`)	2024-06-10 12:41:51 +02:00
Andrii Chubatiuk	1b49413d4c	removed exclusion of recording label in alert expression, which is utilized in alert annotations (#6441 ) ### Describe Your Changes Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6439 ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit `ff458af25e`)	2024-06-10 12:41:48 +02:00
Dmytro Kozlov	a4bdc14bc5	vmctl: disable progress bar for prometheus snapshot migrations (#6385 ) * deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar` * new `--disable-progress-bar` consistently disables usage of progress bar for all migration modes. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8b46bb0c41`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-10 12:41:44 +02:00
Hui Wang	028a80613f	lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338 ) …pAuth.* address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329, makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey` behavior the same way, but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still protected by httpAuth.*. All the available key are listed in https://docs.victoriametrics.com/single-server-victoriametrics/#security. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `61dce6f2a1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-10 12:41:29 +02:00
Andrii Chubatiuk	6fd314d8ba	vmagent: updated dashboard and alert for stream aggregation (#6427 ) ### Describe Your Changes Added streaming aggregation section to vmagent dashboards Added alert for streaming aggregation and deduplication flush timeouts Removed deprecated compose versions from compose files Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2da45a8368`)	2024-06-10 12:37:22 +02:00
Artem Navoiev	d6b56a1460	docs victorialogs: add alias for quickstart for VL to be consistent with other quickstarts Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `318e9e9de0`)	2024-06-10 12:37:19 +02:00
Artem Navoiev	a383e00e48	add alias for vmalert-tools Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `8f254232bb`)	2024-06-10 12:37:16 +02:00
Aliaksandr Valialkin	792dd697f4	docs/CHANGELOG.md: document v1.93.15 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.15	2024-06-07 23:43:20 +02:00
Aliaksandr Valialkin	f61bf790a4	docs/LTS-releases.md: update v1.97.4 LTS release to v1.97.5 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5	2024-06-07 20:16:55 +02:00
Aliaksandr Valialkin	37ca3a2b0c	docs/CHANGELOG.md: add changelog for v1.97.5 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5	2024-06-07 20:03:35 +02:00
Aliaksandr Valialkin	3f883559e2	docs/CHANGELOG.md: cut v1.202.0-rc1 release	2024-06-07 16:53:48 +02:00
Aliaksandr Valialkin	7de3019f2b	README.md: sync with docs/Cluster-VictoriaMetrics.md after `a6655322b1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362	2024-06-07 16:49:56 +02:00
Aliaksandr Valialkin	a38759f66a	docs: run `make docs-sync` after `c740a8042e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362	2024-06-07 16:45:53 +02:00
Aliaksandr Valialkin	abb71b7dfc	vendor: run `make vendor-update`	2024-06-07 16:40:50 +02:00
Aliaksandr Valialkin	3492f4e1fe	app/vmselect/vmui: run `make vmui-update` after c236e3c03c1bf8ca00292b800a839fcb300e7e51 and 04744c274c269f6b6efb45f68df11abe0fb0ce25	2024-06-07 16:39:06 +02:00
Aliaksandr Valialkin	32aa0751a1	lib/streamaggr: follow-up for `7cb894a777` - Use bytesutil.InternString() instead of strings.Clone() for inputKey and outputKey in aggregatorpushSamples(). This should reduce string allocation rate, since strings can be re-used between aggrState flushes. - Reduce memory allocations at dedupAggrShard by storing dedupAggrSample by value in the active series map. - Remove duplicate call to bytesutil.InternBytes() at Deduplicator, since it is already called inside dedupAggr.pushSamples(). - Add missing string interning at rateAggrState.pushSamples(). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402	2024-06-07 16:35:53 +02:00
Roman Khavronenko	78121642df	lib/streamaggr: reduce number of inuse objects (#6402 ) The main change is getting rid of interning of sample key. It was discovered that for cases with many unique time series aggregated by vmagent interned keys could grow up to hundreds of millions of objects. This has negative impact on the following aspects: 1. It slows down garbage collection cycles, as GC has to scan all inuse objects periodically. The higher is the number of inuse objects, the longer it takes/the more CPU it takes. 2. It slows down the hot path of samples aggregation where each key needs to be looked up in the map first. The change makes code more fragile, but suppose to provide performance optimization for heavy-loaded vmagents with stream aggregation enabled. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-06-07 16:35:52 +02:00
Roman Khavronenko	fae589bb83	lib/promrelabel: speedup label match by `__name__` (#6432 ) The change adds a fastpath for `equalValue` comparisons against `__name__` label by avoiding calls to `toCanonicalLabelName` func. This speedups matches by metric name like `'foo'`. See bench stats below: ``` benchcmp old.txt new.txt benchmark old ns/op new ns/op delta BenchmarkIfExpression/equal_label:_last-10 35.6 35.1 -1.18% BenchmarkIfExpression/equal_label:_middle-10 18.3 17.3 -5.41% BenchmarkIfExpression/equal_label:_first-10 1.20 1.24 +2.74% BenchmarkIfExpression/equal___name__:_last-10 10.1 4.96 -50.75% BenchmarkIfExpression/equal___name__:_middle-10 5.79 3.16 -45.41% BenchmarkIfExpression/equal___name__:_first-10 1.17 1.05 -9.76% ``` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-07 16:35:52 +02:00
Andrii Chubatiuk	93cd08f15f	lib/streamaggr: metrics to track dropped, nan samples and samples lag (#6358 ) ### Describe Your Changes Added streamaggr metrics to: - `vm_streamaggr_samples_lag_seconds` - samples lag - `vm_streamaggr_ignored_samples_total{reason="nan"}` - ignored NaN samples - `vm_streamaggr_ignored_samples_total{reason="too_old"}` - ignored old samples (cherry picked from commit `185fac03b3`)	2024-06-06 19:22:45 +02:00
Aliaksandr Valialkin	53382ae837	lib/logstorage: work-in-progress	2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin	6d986f731b	docs/VictoriaLogs/CHANGELOG.md: document the `a68c2c0f17`	2024-06-06 12:21:47 +02:00
Aliaksandr Valialkin	9dfc7190fe	app/vlselect/vmui: run `make vmui-logs-update` after `a68c2c0f17` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407	2024-06-06 12:21:47 +02:00
Yury Molodov	8cf417e1c7	vmui/logs: improve log display for group view (#6419 ) ### Describe Your Changes 1) Set the default limit to `50`. #6408 2) Configure the default search to cover the `last 5 minutes` and include all messages (``). #6405 3) In the header, display only streams and group by stream. #6406 4) Add log processing, without the fields `msg`, `time`, and `stream`. 5) When clicking on logs, display a list of all fields. #6407 <img width="400" alt="image" src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/666dcaa3-20fb-4828-b77b-1d849dd9a8ed"> ### Checklist The following checks are mandatory*: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-06 12:21:46 +02:00
Dima Lazerka	362ee240cd	vmui: Improve DownloadConfig button interaction with VMAnomaly (#6397 ) Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>	2024-06-06 12:12:58 +02:00
Github Actions	2e3c039113	Automatic update operator docs from VictoriaMetrics/operator@32f7674 (#6414 )	2024-06-06 12:12:58 +02:00
Aliaksandr Valialkin	c6b6900152	docs/VictoriaLogs/LogsQL.md: typo fix: replace `from _json` with `from _msg` where appropriate	2024-06-05 10:15:56 +02:00
Aliaksandr Valialkin	be4aefd133	docs/VictoriaLogs/LogsQL.md: substitute TAG with APP-NAME[PROCID] in rfc3164 syslog message format at `unpack_syslog` pipe docs	2024-06-05 10:13:58 +02:00
Aliaksandr Valialkin	a200fb433a	lib/logstorage: allow using `eval` keyword instead of `math` keyword in `math` pipe	2024-06-05 10:08:08 +02:00
Aliaksandr Valialkin	f8336783c8	docs/VictoriaLogs/LogsQL.md: typo fix: substitute "'" with "`" in front of` John`	2024-06-05 09:51:29 +02:00
Aliaksandr Valialkin	9e79a632c1	docs/VictoriaLogs/LogsQL.md: typo fix: its -> it	2024-06-05 09:46:35 +02:00
Zhu Jiekun	024ab786ce	docs: [deployment] update CHANGELOG.md to include go version change 1.22.4 (#6412 ) ### Describe Your Changes Update CHANGELOG to include go version change introduced in `43cf221681` Also see: - https://go.dev/doc/devel/release - https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved ### Checklist The following checks are mandatory: - [X] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-06-05 09:46:34 +02:00
Aliaksandr Valialkin	b45e466a1b	lib/logstorage: work-in-progress	2024-06-05 03:18:25 +02:00
hagen1778	607007add9	docs: add technical articles by @jiekun Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-05 03:16:41 +02:00
pludov	2efd97a63c	lib/fs: support NFS implementations that return EEXIST instead of ENOTEMPTY (#6398 ) ### Describe Your Changes Fix for issue #6396: according to rmdir manpage, ENOTEMPTY and EEXIST should be treated equally https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396 ### Checklist The following checks are mandatory: - [x ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Co-authored-by: Ludovic Pollet <ludovic.pollet@exfo.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3ddae77c63`)	2024-06-04 15:30:48 +02:00
Aliaksandr Valialkin	1a1ee87b46	docs/VictoriaLogs/LogsQL.md: mention unpack_syslog pipe in the Transformations section	2024-06-04 03:17:14 +02:00
Aliaksandr Valialkin	1ce8a9a751	lib/logstorage: allow typing `asc` in `sort` pipe for the sake of consistency with `desc`	2024-06-04 02:29:18 +02:00
Aliaksandr Valialkin	7d31574071	docs/VictoriaLogs/Roadmap.md: remove LogsQL transformations from roadmap, since they are already implemented	2024-06-04 02:21:23 +02:00
Aliaksandr Valialkin	39d238592c	docs/VictoriaLogs/logsql-examples.md: typo fix	2024-06-04 02:17:59 +02:00
Aliaksandr Valialkin	53928a44d1	docs/VictoriaLogs/logsql-examples.md: typo fix	2024-06-04 02:10:56 +02:00
Aliaksandr Valialkin	b7b3a9e9a3	lib/logstorage: work-in-progress	2024-06-04 01:50:55 +02:00
hagen1778	d6096a477f	app/vmalert: rm extra response for unsupported path Unsupported path is already handled by `lib/httpserver`. This prevents from misleading errors in logs caused by double-writing response headers. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a5f81f67fd`)	2024-06-03 12:53:38 +02:00
Hui Wang	39286afae3	vmalert-tool: fix float values template in `input_series` (#6395 ) address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6391 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e3e40cb848`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-03 11:53:58 +02:00
Zakhar Bessarab	ff1bf76237	deployment/docker: add scratch-based images (#6386 ) ### Describe Your Changes Scratch based images will be using a separate tag: "(version)-scratch" and will be built for the same architecture as regular images. This is useful for environments with higher security standards. In this case using alpine as base layer requires updating images more frequently in order to get the latest updates for the base image, even in case the user did not need to update VictoriaMetrics version. Tested that scratch images work for: - vmagent - enterprise with kafka and opensource - cluster - single-node No issues observed so far. cc: @tenmozes ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `7dc9124ba7`)	2024-06-03 11:53:45 +02:00
Artem Navoiev	ea2b32d2d4	deployment docker: use line formatting in alerts-health fixes #6393 (#6394 ) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `508946ed9d`)	2024-06-03 11:53:42 +02:00
Github Actions	32a9c62481	Automatic update operator docs from VictoriaMetrics/operator@d936bb7 (#6378 ) (cherry picked from commit `53422797a7`)	2024-06-03 11:53:39 +02:00
hagen1778	73c9981335	chore: follow-up after `c740a8042e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6d8e02f278`)	2024-06-03 11:53:37 +02:00
Arkadii Yakovets	a6655322b1	docs: fix docs/ and README.md spelling errors (#6362 ) Fixes `docs/` and `README.md` typos and errors. Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com> (cherry picked from commit `c740a8042e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-06-03 11:53:33 +02:00
Nikolay	908a50f79d	app/vmalert: adds idleConnTimeout flags and retry trivial network errors (#6382 ) * ".idleConnTimeout" flags must reduce probability of `write: broken pipe` and `read: connection reset by peer` errors Those errors may occur if remote server closes TCP socket for connection, while it's still exist at client. single time retries for `write: broken pipe` and `read: connection reset by peer` must handle a case for incorrectly configured timeouts at middleware proxies, mitigate minor network issues. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661 ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. --------- Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `b97916276f`)	2024-06-03 11:52:58 +02:00
Aliaksandr Valialkin	b994e64567	docs/VictoriaLogs: typo fixes	2024-05-30 16:35:25 +02:00
Aliaksandr Valialkin	2fd8cf15c4	deployment/docker: update VictoriaLogs Docker image from v0.14.0-victorialogs to v0.15.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.15.0-victorialogs	2024-05-30 16:27:53 +02:00
Aliaksandr Valialkin	540bbb63a2	lib/logstorage: work-in-progress	2024-05-30 16:19:36 +02:00
Zakhar Bessarab	b08ee34061	docs/metricsql: fix typo in link (#6384 ) ### Describe Your Changes Fix typo in link. ### Checklist The following checks are mandatory: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-05-30 16:17:20 +02:00
Fred Navruzov	b200c4263a	docs/vmanomaly: improve FAQ (#6369 ) ### Describe Your Change More explicit [vmanomaly FAQ](https://docs.victoriametrics.com/anomaly-detection/faq/index.html), based on common Q&A from recent communications with users ### Checklist The following checks are mandatory: - ✔️ My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).	2024-05-30 16:16:56 +02:00
Dmytro Kozlov	617ec1fbec	docs: update managed create deployment description (#6376 ) ### Describe Your Changes Updated description how to create deployment in the cloud soulution	2024-05-30 16:16:18 +02:00
yumeiyin	95b8cf76f8	chore: remove redundant words (#6348 ) (cherry picked from commit `9289c7512d`)	2024-05-29 14:37:04 +02:00
Roman Khavronenko	189af53142	lib/storage: filter deleted label names and values from `/api/v1/labe… (#6342 ) …ls` and `/api/v1/label/.../values` Check for deleted metrics when `match[]` filter matches small number of time series (optimized path). The issue was introduced [v1.81.0](https://docs.victoriametrics.com/changelog_2022/#v1810). Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6300 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b984f4672e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-29 14:37:00 +02:00
Andrii Chubatiuk	2c4a42554a	app/vmagent: fixed streamaggr args (#6374 ) use GetOptionalArg instead of index to fallback to a first argument if index is absent for remotewrite.streamaggr.config (cherry picked from commit `7e5a206057`)	2024-05-29 14:04:24 +02:00
Github Actions	9d301809b6	Automatic update operator docs from VictoriaMetrics/operator@cf753a6 (#6373 ) (cherry picked from commit `14ba7b237d`)	2024-05-29 14:04:24 +02:00
Aliaksandr Valialkin	e83fd4a117	lib/logstorage: work-in-progress	2024-05-29 01:52:34 +02:00
Aliaksandr Valialkin	6ddc618731	docs/VictoriaLogs: typo fixes	2024-05-28 22:48:57 +02:00
Aliaksandr Valialkin	2395c7e65e	deployment/docker: update VictoriaLogs image from v0.12.1-victorialogs to v0.13.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.13.0-victorialogs	2024-05-28 22:16:30 +02:00
Alexander Marshalov	7d532a31fb	Update base Alpine image to 3.20.0 to avoid security risks (#6370 ) fixes: CVE-2023-42366, CVE-2023-42363, CVE-2024-4603, CVE-2024-2511, CVE-2024-24788, CVE-2024-24787	2024-05-28 22:16:29 +02:00
Aliaksandr Valialkin	79c03fc35f	lib/logstorage: work-in-progress	2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin	d429c0b83e	deployment: update VictoriaLogs image from v0.12.0-victorialogs to v0.12.1-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.12.1-victorialogs	2024-05-26 02:08:52 +02:00
Aliaksandr Valialkin	ce5e4c842a	lib/logstorage: fix golangci-lint warnings	2024-05-26 02:02:41 +02:00
Aliaksandr Valialkin	afa597ce2a	lib/logstorage: work-in-progress	2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin	6427b3c3c0	lib/logstorage: work-in-progress	2024-05-25 22:59:21 +02:00
Aliaksandr Valialkin	9edbeca46b	lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll	2024-05-25 22:13:44 +02:00
Aliaksandr Valialkin	f02c5cb79a	docs/VictoriaLogs/CHANGELOG.md: cut v0.12.0-victorialogs	2024-05-25 21:40:20 +02:00
Aliaksandr Valialkin	03fe4c8963	lib/logstorage: work-in-progress	2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin	3152df2bce	lib/logstorage: work-in-progress	2024-05-25 00:31:55 +02:00
hagen1778	63b83d62e8	deployment/alerts: add new alerting rules `TooLongLabelValues` and `TooLongLabelNames` to notify about truncation of label values or names respectively. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1be1e9a7a4`)	2024-05-24 16:08:41 +02:00
hagen1778	385500f8f5	deployment/alerts: rm `ProcessNearFDLimits` alert from alerts-cluster As it is already present in alerts-health file Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `49f13b12d9`)	2024-05-24 16:08:39 +02:00
Nikolay	5025ede7bc	lib/mergeset: adds tracking for indexdb records drop (#6297 ) It allows to create alert for possible item drops at indexdb. It may happen, if ingested metric size exceeds max indexdb item size. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `69d244e6fb`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-24 16:08:34 +02:00
Zakhar Bessarab	85eee7de0a	app/vmselect: update flag description (#6347 ) Update wording to highlight that cache is not persistent if flag is value is empty. Previously, it was not clear if cache is not used at all or just not persistent.	2024-05-24 15:58:54 +02:00
Aliaksandr Valialkin	db48642302	deployment: update Docker image tag for VictoriaLogs from v0.8.0-victorialogs to v0.10.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.10.0-victorialogs	2024-05-24 03:16:06 +02:00
Aliaksandr Valialkin	e5ca4fff76	docs/VictoriaLogs/CHANGELOG.md: cut v0.10.0-victorialogs	2024-05-24 03:08:29 +02:00
Aliaksandr Valialkin	7a2a2f173e	lib/logstorage: work-in-progress	2024-05-24 03:07:07 +02:00
Github Actions	8025cfdb73	Automatic update operator docs from VictoriaMetrics/operator@4b059a4 (#6332 )	2024-05-24 03:05:39 +02:00
hagen1778	d691d515f1	docs: fix changelog formatting Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-24 03:05:39 +02:00
Aliaksandr Valialkin	ae8b2bcf2e	app/vlselect: fix loading web UI	2024-05-22 23:25:00 +02:00
Aliaksandr Valialkin	2befafa264	docs/VictoriaLogs/LogsQL.md: mention that it is better from performance PoV parsing JSON and logfmt logs before the ingestion into VictoriaLogs	2024-05-22 23:04:48 +02:00
Aliaksandr Valialkin	181b5c5437	docs/VictoriaLogs/LogsQL.md: typo fixes	2024-05-22 22:24:56 +02:00
Aliaksandr Valialkin	802d7edd0e	docs/VictoriaLogs/CHANGELOG.md: cut v0.8.0-victorialogs	2024-05-22 22:07:38 +02:00
Aliaksandr Valialkin	6addc79bdb	app/vlselect/vmui: run `make vmui-logs-update`	2024-05-22 22:06:28 +02:00
Aliaksandr Valialkin	80c4fad8a9	vendor: run `make vendor-update`	2024-05-22 21:58:53 +02:00
Nikolay	dfbd2f8ff7	lib/storage: change default value for maxLabelValueLen to 1024 (#6313 ) * It must reduce memory usage for misbehaving clients. Since VictoriaMetrics stores sparse index inmemory. * Reduce disk space usage for indexdb. * Prevent possible indexDB items drops. * It may trigger slow insert and new timeseries registration due to default value for flag change https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6176 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-05-22 21:55:21 +02:00
Alexander Marshalov	0b70c4c1f1	[vmlogs] fixed time parsing with millisecond precision time (#6293 ) (#6295 ) fix for #6293 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-05-22 21:54:50 +02:00
Yury Molodov	252a196405	vmui/logs: fix parsing long `_msg` values (#6310 ) This PR fixes an issue where parsing long `_msg` values caused errors, resulting in some log records not being displayed. The error occurred due to partial processing of strings. In some cases, a long record could be split into multiple chunks, causing only part of the record to be processed instead of the entire entry. #6281 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-05-22 21:44:49 +02:00
Aliaksandr Valialkin	04d0dd2542	lib/logstorage: work-in-progress	2024-05-22 21:01:28 +02:00
hagen1778	89819f2054	dashboards: use `$__interval` variable for offsets and look-behind windows in annotations This should improve precision of `restarts` and `version change` annotations when zooming-in/zooming-out on the dashboards. The change also makes `restarts` dashboard visible on the panels, so user can disable it from displaying if needed. This could be useful when restarts overlap with version change events. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9dd9b4442f`)	2024-05-22 16:40:08 +02:00
Roman Khavronenko	f3e893f699	lib/backup: add `-s3TLSInsecureSkipVerify` command-line flag (#6318 ) * The new flag can be used for for skipping TLS certificates verification when connecting to S3 endpoint. Affects vmbackup, vmrestore, vmbackupmanager. * replace deprecated `EndpointResolver` with `BaseEndpoint` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1056 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ac836bcf6c`)	2024-05-22 16:40:06 +02:00
Daria Karavaieva	aea042bc9d	docs/vmanomaly: backtest parameter fix (#6327 ) ### Fix backtest parameters Fix date parameters names in vmanomaly backtest scheduler component documentation. (cherry picked from commit `c40f355496`)	2024-05-22 16:40:01 +02:00
Hui Wang	5b8c3fc9d0	app/vmalert: support DNS SRV record in `-remoteWrite.url` (#6299 ) part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053, supports [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) address in `-remoteWrite.url` command-line option. (cherry picked from commit `d7b5062917`)	2024-05-22 10:53:22 +02:00
Hui Wang	0b9e59e67e	vmalert: speed up reloading rules from object storage by verifying ob… (#755 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6210 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `974b7783ee`)	2024-05-21 16:37:27 +02:00
Github Actions	26306fec80	Automatic update operator docs from VictoriaMetrics/operator@84833cc (#6316 ) (cherry picked from commit `23619a3adf`)	2024-05-21 16:37:25 +02:00
hagen1778	0dd3fec2b7	deployment/dashboards: fix `AnnotationQueryRunner` error in Grafana The error appears when executing annotations query against Prometheus backend because the query itself hasn't specified look-behind window (which is allowed in VictoriaMetrics query engine). https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6309 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c746ba154d`)	2024-05-21 16:37:23 +02:00
Artem Navoiev	f4743dcef2	docs: victorialogs mention that our bench suite can be run against Loki as well Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `cee4bfebd7`)	2024-05-21 16:37:21 +02:00
Yury Molodov	33eaa18c14	vmui: fix URL params handling for navigation (#6284 ) This PR fixes the handling of URL parameters to ensure correct browser navigation using the back and forward buttons. #6126 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232 (cherry picked from commit `f14497f1cd`)	2024-05-20 14:46:41 +02:00
Yury Molodov	97c3c946a7	vmui/logs: change time range to `start` and `end` query args (#6296 ) change time range limitation from `_time` in the expression to `start` and `end` query args. (cherry picked from commit `a6a599cbdc`)	2024-05-20 14:46:39 +02:00
Roman Khavronenko	3e8b5e74d5	lib/streamaggr: skip empty aggregators (#6307 ) Prevent excessive resource usage when stream aggregation config file contains no matchers by prevent pushing data into Aggregators object. Before this change a lot of extra work was invoked without reason. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `7ce052b32d`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-20 14:46:36 +02:00
Roman Khavronenko	8daa1d9505	app/vmagent: fix panic on shutdown when no global deduplication is co… (#6308 ) …nfigured Follow-up for `f153f54d11` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `7dc18bf67a`)	2024-05-20 14:46:10 +02:00
Aliaksandr Valialkin	45fbcc74e0	lib/logstorage: fix golangci-lint warnings	2024-05-20 11:04:37 +02:00
Github Actions	d86d768aec	Automatic update operator docs from VictoriaMetrics/operator@c8534a5 (#6305 )	2024-05-20 11:04:36 +02:00
Aliaksandr Valialkin	10dd940c54	docs/VictoriaLogs: typo fixes	2024-05-20 04:48:42 +02:00
Aliaksandr Valialkin	18982e1ab7	docs/VictoriaLogs/LogsQL.md: typo fix	2024-05-20 04:27:44 +02:00
Aliaksandr Valialkin	b02d085a52	docs/VictoriaLogs/LogsQL.md: typo fix	2024-05-20 04:24:34 +02:00
Aliaksandr Valialkin	400f43dd48	deployment: update VictoriaLogs from v0.6.0-victorialogs to v0.8.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.8.0-victorialogs	2024-05-20 04:17:51 +02:00
Aliaksandr Valialkin	582e7d5439	lib/logstorage: work-in-progress	2024-05-20 04:09:15 +02:00
viperstars	ab78f3c89d	app/vmagent/remotewrite: skip sending empty block to downstream server (#6241 ) Occasionally, vmagent sends empty blocks to downstream servers. If a downstream server returns an unexpected response, vmagent gets stuck in a retry loop. While vmagent handles 400 and 409 errors, there are various prometheus remote write implementations that return different error codes. For example, vector returns a 422 error. To mitigate the risk of vmagent getting stuck in a retry loop, it is advisable to skip sending empty blocks to downstream servers. Co-authored-by: hao.peng <hao.peng@smartx.com> Co-authored-by: Zhu Jiekun <jiekun.dev@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3661373cc2`)	2024-05-17 14:57:07 +02:00
Andrii Chubatiuk	23c7193ca4	docs: added information about difference between default_rollup and last_over_time (#6282 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6279 (cherry picked from commit `ce3cfd720e`)	2024-05-17 14:57:04 +02:00
Yury Molodov	5bfbfe6ad2	vmui: remove redundant requests on the `Explore Cardinality` page (#6263 ) Remove redundant requests on the Explore Cardinality page. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240 (cherry picked from commit `be291c36f7`)	2024-05-17 14:56:55 +02:00
Yury Molodov	0edef9105b	vmui: fix calendar display (#6255 ) Fix the calendar display issue occurring with the `UTC+00:00` timezone https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239 (cherry picked from commit `4ad577cc6f`)	2024-05-17 14:56:53 +02:00
Andrii Chubatiuk	fe332c3419	app/vmagent: add global aggregator (#6268 ) Add global stream aggregation for VMAgent https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5467 (cherry picked from commit `f153f54d11`)	2024-05-17 14:01:31 +02:00
Nikolay	ee4a94a371	follow-up for `c6c5a5a186` (#6265 ) * adds datadog extensions for statsd: - multiple packed values (v1.1) - additional types distribution, histogram * adds type check and append metric type to the labels with special tag name `__statsd_metric_type__`. It simplifies streaming aggregation config. * remove statsd support from cluster, since cluster doesn't support stream aggregation. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b2765c45d0`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-17 13:49:24 +02:00
Roman Khavronenko	a5c427bac4	app/vmalert/datasource: reduce number of allocations when parsing instant responses (#6272 ) Allocations are reduced by implementing custom json parser via fastjson lib. The change also re-uses `promInstant` object in attempt to reduce number of allocations when parsing big responses, as usually happens with heavy recording rules. ``` name old allocs/op new allocs/op delta ParsePrometheusResponse/Instant-10 9.65k ± 0% 5.60k ± 0% ~ (p=1.000 n=1+1) ``` Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4f0525852f`)	2024-05-16 09:35:58 +02:00
Aliaksandr Valialkin	473c7a344c	docs/VictoriaLogs/CHANGELOG.md: cut v0.7.0-victorialogs (cherry picked from commit `6fdba8599d`)	2024-05-16 09:35:57 +02:00
Aliaksandr Valialkin	28626db066	lib/logstorage: work-in-progress (cherry picked from commit `0aa19a2837`)	2024-05-16 09:35:55 +02:00
Aliaksandr Valialkin	5dbc4ad5ef	lib/streamaggr: properly return output key from getOutputKey The bug has been introduced in `cc2647d212` (cherry picked from commit `b617dc9c0b`)	2024-05-16 09:35:53 +02:00
Roman Khavronenko	955d36357c	app/vmalert/rule: reduce number of allocations for getStaleSeries fn (#6269 ) Allocations are reduced by re-using the byte buffer when converting labels to string keys. ``` name old allocs/op new allocs/op delta GetStaleSeries-10 703 ± 0% 203 ± 0% ~ (p=1.000 n=1+1) ``` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b0c1f3d819`)	2024-05-16 09:35:51 +02:00
Nikolay	2b2fdffd77	app/vmauth: explicitly unregister metrics set for auth config (#6252 ) it's needed to remove Summary metric type from the global state of metrics package. metrics package tracks each bucket of summary and periodically swaps old buckets with new. Simple set unregister is not enough to release memory used by Set https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6247 (cherry picked from commit `6a6e34ab8e`)	2024-05-14 09:28:37 +02:00
Aliaksandr Valialkin	8af6c80bf2	docs/VictoriaLogs/CHANGELOG.md: cut v0.6.1-victorialogs	2024-05-14 03:07:02 +02:00
Aliaksandr Valialkin	b1ee7bca1a	lib/logstorage: work-in-progress	2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin	f52275bbd7	lib/logstorage: work-in-progress Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258	2024-05-14 01:49:58 +02:00
Aliaksandr Valialkin	207b4bd91d	lib/storage: fix SearchQuery.Unmarshal() after `32193b6059`	2024-05-14 01:39:01 +02:00
Aliaksandr Valialkin	32193b6059	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit Change the return values for these functions - now they return the unmarshaled result plus the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling. This improves performance of these functions in hot loops of VictoriaLogs a bit.	2024-05-14 01:30:25 +02:00
Aliaksandr Valialkin	2e12119a9e	lib/stringsutil: add LessNatural() function for natural sorting Natural sorting is needed for sort_by_label_natural() and sort_by_label_natural_desc() functions in MetricsQL - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6192 and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6256 Natural sorting will be also used by `\| sort ...` pipe in VictoriaLogs - see https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe (cherry picked from commit `707f3a69db`)	2024-05-13 17:08:56 +02:00
Hui Wang	ec56f4625e	storage: correctly apply `-inmemoryDataFlushInterval` when it's set t… (#6221 ) …o minimum supported value 1s pendingRowsFlushInterval was bumped to 2s in `73f0a805e2` (cherry picked from commit `4c80b17027`)	2024-05-13 16:50:02 +02:00
Andrii Chubatiuk	b9eb527d98	lib/streamaggr: added rate_sum and rate_avg to benchmarks, lint fix (#6264 ) fixed lint for rate outputs (cherry picked from commit `ce25d68b45`)	2024-05-13 16:49:59 +02:00
Andrii Chubatiuk	d9cddf1ad8	lib/streamaggr: added rate and rate_avg output (#6243 ) Added `rate` and `rate_avg` output Resource usage is the same as for increase output, tested on a benchmark --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9c3d44c8c9`)	2024-05-13 16:49:39 +02:00
hagen1778	84a896cd6e	lib/logstorage: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `17283fab6c`)	2024-05-13 16:49:37 +02:00
Andrii Chubatiuk	ec2273b247	app/vmagent: removed deprecated -remoteWrite.multitenantURL flag support (#6253 ) Removed deprecated `-remoteWrite.multitenantURL` flag to simplify global stream aggregation --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `680b8c25c8`)	2024-05-13 16:49:33 +02:00
Yury Molodov	f18ae015de	vmui/vmanomaly: add download config button (#6231 ) This pull request adds a button to the vmanomaly ui that opens a modal window for viewing and downloading the config file. <img width="610" alt="button" src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/0132b178-eb73-4272-8144-be7ed2a8dcaf"> <img height="300" alt="error" src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/6d9f2627-77d7-4ce6-b73b-542ce1bbc999"> <img height="300" alt="modal" src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/680bffdd-d6a3-445e-bd48-8f0feb30016e"> (cherry picked from commit `37c22ee053`)	2024-05-13 16:49:31 +02:00
Yury Molodov	e430ab1999	vmui/vmanomaly: fix default server url (#6178 ) This PR for ui vmanomaly eliminates URL parameters to automatically use the default server URL, simplifying URLs like: From http://localhost:3000/#/?g0.expr=vm_blocks... to http://localhost:3000 From http://localhost:3000/select/0/vmui/#/?g0.expr=vm_blocks... to http://localhost:3000/select/0/vmui/ etc. (cherry picked from commit `29bd120126`)	2024-05-13 16:49:29 +02:00
Aliaksandr Valialkin	3c645c4a79	deployment: update VictoriaLogs Docker image from v0.5.2-victorialogs to v0.6.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.6.0-victorialogs	2024-05-12 23:23:05 +02:00
Aliaksandr Valialkin	d341cd5db1	deployment/docker/Makefile: group app-via-docker-* and package-via-docker-* rules with CGO_ENABLED=1 together for better maintainability	2024-05-12 23:11:54 +02:00
Aliaksandr Valialkin	ebe1ede42e	deployment/docker/Makefile: rename EXTRA_ENVS to EXTRA_DOCKER_ENVS The purpose of EXTRA_DOCKER_ENVS name is more clear than EXTRA_ENVS. While at it, make the following small fixes: - Pass GOARM=5 to Docker builder when building Docker packages for GOARCH=arm in the same way it is passed to the builder when building production binaries for GOARCH=arm. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965 - Set GCO_ENABLED=1 for package-via-docker-amd64, which has been accidentally removed in `07496d7d92` - Consistently use 'CGO_ENABLED=... GOARCH=...' order of env vars at package-via-docker-, because this order is used in app-via-docker- Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6158 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6172 This is a follow-up for `07496d7d92` and `7958f38864`	2024-05-12 23:11:52 +02:00
Aliaksandr Valialkin	b24daf7712	docs/VictoriaLogs/LogsQL.md: cross-reference uniq pipe with uniq_values stats function	2024-05-12 16:48:12 +02:00
Aliaksandr Valialkin	3ba65c5804	docs/VictoriaLogs/CHANGELOG.md: cut v0.6.0-victorialogs	2024-05-12 16:38:09 +02:00
Aliaksandr Valialkin	147704aab0	lib/logstorage: initial implementation of pipes in LogsQL See https://docs.victoriametrics.com/victorialogs/logsql/#pipes	2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin	9dc9c892b7	lib/encoding: optimizing UnmarshalVarUint64 and UnmarshalVarInt64 a bit	2024-05-12 16:35:24 +02:00
Aliaksandr Valialkin	1bca5e37c1	vendor: run `make vendor-update`	2024-05-12 16:18:53 +02:00
Aliaksandr Valialkin	817caae3f6	go.mod: update the required Go version from 1.21 to 1.22 This is a follow-up for `95222b2079`	2024-05-12 16:08:39 +02:00
Aliaksandr Valialkin	0d55ef3658	docs/vmauth.md: small fixes after proofreading	2024-05-12 12:35:32 +02:00
Aliaksandr Valialkin	87338633b1	lib/slicesutil: add helper functions for setting slice length and extending its capacity The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.	2024-05-12 11:33:49 +02:00
Aliaksandr Valialkin	9607902289	lib/storage: remove outdated misleading comments	2024-05-12 10:25:06 +02:00
Aliaksandr Valialkin	6b81441ed0	app/vmselect: use strings.EqualFold instead of strings.ToLower where appropriate Strings.EqualFold doesn't allocate memory contrary to strings.ToLower if the input string contains uppercase chars	2024-05-12 10:21:24 +02:00
Aliaksandr Valialkin	536d87cd51	app/vmselect/promql: properly estimate the needed amounts of memory for executing aggregate function over rollup function in incremental mode Incremental aggregation processes only GOMAXPROCS time series at a time, so its' memory usage doesn't depend on the number of input time series. The issue has been introduced in `5138eaeea0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203	2024-05-12 10:14:27 +02:00
Aliaksandr Valialkin	65128ea814	README.md: mention that `-tlsCertFile` and `-tlsKeyFile` options aren't needed when automatic issuing of tls certificates is enabled with `-tlsAutoCertHosts` flag	2024-05-12 09:49:16 +02:00
Aliaksandr Valialkin	0c89adf589	docs/Single-server-VictoriaMetrics.md and docs/README.md: sync with README.md with `make docs-sync` after the commit `c6c5a5a186`	2024-05-12 09:41:39 +02:00
Roman Khavronenko	0bed453737	Feature allow configuring disableOnDiskQueue and dropSamplesOnOverload per url (#6248 ) * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): allow configuring `-remoteWrite.disableOnDiskQueue` and `-remoteWrite.dropSamplesOnOverload` cmd-line flags per each `-remoteWrite.url`. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065). Thanks to @rbizos for implementaion! * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add labels `path` and `url` to metrics `vmagent_remotewrite_push_failures_total` and `vmagent_remotewrite_samples_dropped_total`. Now number of failed pushes and dropped samples can be tracked per `-remoteWrite.url`. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Raphael Bizos <r.bizos@criteo.com> (cherry picked from commit `87fd400dfc`)	2024-05-10 14:32:23 +02:00
Github Actions	ab5b233c3a	Automatic update operator docs from VictoriaMetrics/operator@0829591 (#6250 ) (cherry picked from commit `c87ce86d96`)	2024-05-10 14:32:18 +02:00
qiangxuhui	885fc4122a	Add build support for loong64 (#6222 ) ### Describe Your Changes Added makefile rule for `GOARCH=loong64` to support building all VictoriaMetrics components on the `loongarch64` platform. ### Checklist The following checks are mandatory: * [X] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: qiangxuhui <qiangxuhui@loongson.cn> (cherry picked from commit `80f3644ee3`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-10 14:32:05 +02:00
Github Actions	ac33ef71ca	Automatic update operator docs from VictoriaMetrics/operator@4a51b37 (#6245 ) (cherry picked from commit `a8d0c1a62d`)	2024-05-10 14:29:22 +02:00
hagen1778	879170221c	app/vmselect/vmui: add missing static files These files weren't added to the git after `make vmui-build vmui-update` command in commit `7fd9325e62 (diff-50d9a4b91bdad190f2db92553736267103ab4225dfb6642b675fb4b8196e6560)` Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6224 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `56531abd56`)	2024-05-10 14:29:20 +02:00
Roman Khavronenko	7be6fcd8fd	lib/streamaggr: set correct suffix `<output>_prometheus` (#6228 ) Set correct suffix `<output>_prometheus` for aggregation outputs `increase_prometheus` and `total_prometheus` Before, outputs `total` and `total_prometheus` or `increase` and `increase_prometheus` had the same suffix. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8a03e987cb`)	2024-05-10 14:29:01 +02:00
Andrii Chubatiuk	f3d65ba902	streamaggr: made labels compressor shared (#6173 ) Though labels compressor is quite resource intensive, each aggregator and deduplicator instance has it's own compressor. Made it shared across all aggregators to consume less resources while using multiple aggregators. Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> (cherry picked from commit `a9283e06a3`)	2024-05-10 14:28:59 +02:00
Zhu Jiekun	139f909cdb	chore: [deployment] upgrade from go 1.22.2 to 1.22.3 to include security fixes (#6238 ) ### Describe Your Changes upgrade from go 1.22.2 to 1.22.3 to include security fixes. Also see: - https://go.dev/doc/devel/release - https://github.com/golang/go/issues?q=milestone%3AGo1.22.3+label%3ACherryPickApproved ### Checklist The following checks are mandatory: - [X] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). Signed-off-by: Jiekun <jiekun.dev@gmail.com> (cherry picked from commit `02851d7800`)	2024-05-10 14:28:56 +02:00
Zhu Jiekun	10063e98a6	feature: [vmagent] Add service discovery support for Vultr (#6068 ) ### Describe Your Changes related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041 #### Added - Added service discovery support for Vultr. #### Docs - `CHANGELOG.md`, `sd_configs.md`, `vmagent.md` are updated. #### Note - Useful links: - Vultr API: https://www.vultr.com/api/#tag/instances/operation/list-instances - Vultr client SDK: https://github.com/vultr/govultr - Prometheus SD: https://github.com/prometheus/prometheus/tree/main/discovery/vultr --- ### Checklist The following checks are mandatory: - [X] I have read the [Contributing Guidelines](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md) - [x] All commits are signed and include `Signed-off-by` line. Use `git commit -s` to include `Signed-off-by` your commits. See this [doc](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) about how to sign your commits. - [x] Tests are passing locally. Use `make test` to run all tests locally. - [x] Linting is passing locally. Use `make check-all` to run all linters locally. Further checks are optional for External Contributions: - [X] Include a link to the GitHub issue in the commit message, if issue exists. - [x] Mention the change in the [Changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md). Explain what has changed and why. If there is a related issue or documentation change - link them as well. Tips for writing a good changelog message:: * Write a human-readable changelog message that describes the problem and solution. * Include a link to the issue or pull request in your changelog message. * Use specific language identifying the fix, such as an error message, metric name, or flag name. * Provide a link to the relevant documentation for any new features you add or modify. - [ ] After your pull request is merged, please add a message to the issue with instructions for how to test the fix or try the feature you added. Here is an [example](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048#issuecomment-1546453726) - [x] Do not close the original issue before the change is released. Please note, in some cases Github can automatically close the issue once PR is merged. Re-open the issue in such case. - [x] If the change somehow affects public interfaces (a new flag was added or updated, or some behavior has changed) - add the corresponding change to documentation. Signed-off-by: Jiekun <jiekun.dev@gmail.com> (cherry picked from commit `17e3d019d2`)	2024-05-10 14:28:51 +02:00
hagen1778	864fbf9125	Statsd protocol compatibility (#5053 ) In this PR I added compatibility with [statsd protocol](https://github.com/b/statsd_spec) with tags to be able to send metrics directly from statsd clients to vmagent or directly to VM. For example its compatible with [statsd-instrument](https://github.com/Shopify/statsd-instrument) and [dogstatsd-ruby](https://github.com/DataDog/dogstatsd-ruby) gems Related issues: #5052, #206, #4600 (cherry picked from commit `c6c5a5a186`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-10 14:28:37 +02:00
Oleg	76af930e4a	Statsd protocol compatibility (#5053 ) In this PR I added compatibility with [statsd protocol](https://github.com/b/statsd_spec) with tags to be able to send metrics directly from statsd clients to vmagent or directly to VM. For example its compatible with [statsd-instrument](https://github.com/Shopify/statsd-instrument) and [dogstatsd-ruby](https://github.com/DataDog/dogstatsd-ruby) gems Related issues: #5052, #206, #4600 (cherry picked from commit `c6c5a5a186`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-05-10 14:27:31 +02:00
Github Actions	cc68484f95	Automatic update operator docs from VictoriaMetrics/operator@2789953 (#6237 ) (cherry picked from commit `55c7dafb35`)	2024-05-10 13:14:24 +02:00
Alexander Marshalov	2922b2a2e0	fix typo in scrape config examples (#6234 ) (cherry picked from commit `3d4988ecf6`)	2024-05-10 13:14:22 +02:00
Github Actions	bb667fd3f7	Automatic update operator docs from VictoriaMetrics/operator@6271553 (#6233 ) (cherry picked from commit `134dcaef33`)	2024-05-10 13:14:19 +02:00
Ted Possible	0206a01d03	Exemplar support (#5982 ) This code adds Exemplars to VMagent and the promscrape parser adhering to OpenMetrics Specifications. This will allow forwarding of exemplars to Prometheus and other third party apps that support OpenMetrics specs. --------- Signed-off-by: Ted Possible <ted_possible@cable.comcast.com> (cherry picked from commit `5a3abfa041`)	2024-05-10 13:14:17 +02:00
hagen1778	33db552a39	docs: mention `influxListenAddr` in URLs format doc Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2561a132ee`)	2024-05-10 13:14:09 +02:00
Andrii Chubatiuk	e26b55db1e	app/vmagent/remotewrite: do not cleanup timeseries which are used in multiple remote write contexts (#6206 ) When at least one remote write has deduplication configured it cleans up timeseries while they can be in use by another remote write without deduplication https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `879771808b`)	2024-05-06 12:10:45 +02:00
Yury Molodov	75af52c1d0	vmui: fix issue preventing first query trace expansion (#6197 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186 (cherry picked from commit `046a4a5ecf`)	2024-04-30 18:39:22 +02:00
hagen1778	798a446d5a	docs: mention actual version in update nodes Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c0050beadc`)	2024-04-30 18:38:20 +02:00
Hui Wang	abd29c15ab	docs: update vmalert and vmagent docs (#6207 ) * restore and actualize doc section explaining duplicated labels error * rm misleading comment about post-aggregation in stream aggregation (cherry picked from commit `e3c226cf92`)	2024-04-30 10:30:19 +02:00
Corporte Gadfly	916170cca2	deployment: minor grammatical fixes in alert descriptions (#6199 ) (cherry picked from commit `8bca4d2de4`)	2024-04-30 10:30:17 +02:00
Roman Khavronenko	fc28390618	app/vmauth: add test for LeastLoaded balance policy (#6144 ) Check if least-loaded works correctly. related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6136 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e2590b339d`)	2024-04-30 10:30:14 +02:00
Zakhar Bessarab	f6eb8912b0	lib/mergeset: improve test coverage (#6118 ) Add test to cover the code path with overflowing shards buffers and triggering merge to partition. This test covers the code path which leaded to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `329c3cbdf0`)	2024-04-30 10:30:12 +02:00
Github Actions	f22b2a0563	Automatic update operator docs from VictoriaMetrics/operator@8f025b3 (#6200 ) Co-authored-by: Hui Wang <haley@victoriametrics.com> (cherry picked from commit `e908effd22`)	2024-04-30 10:30:10 +02:00
hagen1778	d87c8757cf	dashboards: add new panel `Concurrent selects` to `vmstorage` row The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d386a68b59`)	2024-04-30 10:30:08 +02:00
hagen1778	0d77a55961	deployment: update per-tenant-statistic dashboard to be compatible with Grafana 10 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9e18724036`)	2024-04-30 10:30:06 +02:00
hagen1778	a0698d92c3	deployment: update backupmanager dashboard to be compatible with Grafana 10 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5917ac003e`)	2024-04-30 10:30:04 +02:00
hagen1778	40af0fa179	deployment: update operator dashboard to be compatible with Grafana 10 - Use TimeSeries panel instead of deprecated Graph - Update panel styles - Fix version panel Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f0c4d372bd`)	2024-04-30 10:30:02 +02:00
hagen1778	0f72ab8ef6	deployment: bump Grafana version to 10.4.2 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9256df17fa`)	2024-04-30 10:30:00 +02:00
hagen1778	8f48747802	dashboards: add `Network Usage` panel to `Resource Usage` row https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4478 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8606b48ce5`)	2024-04-30 10:29:58 +02:00
Dima Lazerka	d821e13c24	deployment/dashboards: properly show version for non-stable docker images (#6150 ) re: .-(?:tags\|heads)-(.)-(?:0\|dirty)-.* cases: victoria-metrics-20240419-160209-heads-enterprise-single-node-0-g08f933ab0c enterprise-single-node victoria-metrics-20240201-133950-tags-v1.97.1-enterprise-0-g760a8733b v1.97.1-enterprise victoria-metrics-20240419-160209-heads-rotation-part-2-0-ge2367b6d1-dirty-848b54cd rotation-part-2-0-ge2367b6d1 victoria-metrics-20240419-160209-heads-lts-1.93-enterprise-search-contention-0-g30ef4aad21-amd64 lts-1.93-enterprise-search-contention victoria-metrics-20240425-150852-tags-v1.101.0-enterprise-0-g718138c64 v1.101.0-enterprise Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `564463259a`)	2024-04-30 10:29:56 +02:00
Zakhar Bessarab	88677f179c	dashboards/victoria-metrics-single: allow selecting multiple instance values (#5870 ) Allowing to select multiple instance IPs makes it much easier to view metrics for longer periods of time in dynamic environments such as Kubernetes. In k8s update will also cause IP to change making it harder to use dashboard to check the status. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5869 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6b493582da`)	2024-04-30 10:29:54 +02:00
Roman Khavronenko	fac1f6c147	deployment: bump to 1.101.0 (#6194 ) Step 17: > Bump VictoriaMetrics version at deployment/docker/docker-compose.yml and at deployment/docker/docker-compose-cluster.yml. from https://docs.victoriametrics.com/release-guide/ Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5e8c087d42`)	2024-04-30 10:29:32 +02:00
Github Actions	954abe78a3	Automatic update operator docs from VictoriaMetrics/operator@bbf847c (#6189 ) (cherry picked from commit `3a669b4b87`)	2024-04-30 10:29:29 +02:00
hagen1778	310d100ed7	docs/CHANGELOG.md: cut v1.101.0 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5334f0c2ce`)	2024-04-25 16:25:38 +02:00
hagen1778	dfad598092	app/vmselect: run make vmui-update Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `7fd9325e62`)	2024-04-25 16:02:59 +02:00
hagen1778	5b82f615ff	docs: add change for `c0e4ccb7b5` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a62551f773`)	2024-04-25 16:02:57 +02:00
Hui Wang	7fdea4b31c	app/vmselect: implement cmd-line flags `-search.disableImplicitConversions` and `-search.logImplicitConversions` (#6180 ) address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4338 support disable or log [implicit conversions](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions) for subquery with cmd-line flags `-search.disableImplicitConversion` and `-search.logImplicitConversion` Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `dd0d2c77c8`)	2024-04-25 13:08:05 +02:00
Yury Molodov	14c0c06526	vmui: improve error message for server response issues (#6177 ) Updates error messages for better clarity and guidance on server response issues. (cherry picked from commit `57b7d16259`)	2024-04-25 13:08:02 +02:00
Yury Molodov	669cbcb92e	vmui: trigger auto-suggestion at any cursor position (#6155 ) - Implemented auto-suggestion triggers for mid-string cursor positions in vmui. - Improved the suggestion list positioning to appear directly beneath the active text editing area. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5864 (cherry picked from commit `6193fa3dcf`)	2024-04-25 13:08:00 +02:00
hagen1778	d4e901e212	Revert "docs: removed code-style highlighting for commanad-line flags of VM components (#6147 )" This reverts commit `9bedbcfa2f`. (cherry picked from commit `6aaf1768f4`)	2024-04-25 13:07:58 +02:00
Andrii Chubatiuk	8c97dc3b2c	deployment: update makefile package-* targets (#6172 ) Updated package targets in a same manner, how it's done for publish ones in `7958f38864` (cherry picked from commit `07496d7d92`)	2024-04-25 13:07:54 +02:00
hagen1778	6f6e9e1e51	docs/changelog: mention downsampling fixes for ENT version of VM Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2d9bbe1934`)	2024-04-24 17:08:28 +02:00
hagen1778	59b3f21708	Revert "app/vmbackup: introduce new flag type URL (#6152 )" This reverts commit `029060af60`. (cherry picked from commit `679844feaf`)	2024-04-24 17:08:26 +02:00
Roman Khavronenko	ff73b66182	app/vmbackup: introduce new flag type URL (#6152 ) The new flag type is supposed to be used for specifying URL values which could contain sensitive information such as auth tokens in GET params or HTTP basic authentication. The URL flag also allows loading its value from files if `file://` prefix is specified. As example, the new flag type was used in app/vmbackup as it requires specifying `authKey` param for making the snapshot. See related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5973 Thanks to @wasim-nihal for initial implementation https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6060 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `029060af60`)	2024-04-24 17:08:24 +02:00
Github Actions	ff0a1eb3e6	Automatic update operator docs from VictoriaMetrics/operator@6e1a876 (#6179 ) (cherry picked from commit `a14acd4fcc`)	2024-04-24 17:08:21 +02:00
Artem Navoiev	4f07d32548	docs: vmalert remove new lines in configuration section Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `592a9fe9f2`)	2024-04-24 17:08:19 +02:00
Artem Navoiev	f22cfdc616	docs: vmalert remove new lines in configuration section Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `aeba3f39db`)	2024-04-24 17:08:18 +02:00
hagen1778	d4b56d467f	dashboards: show max number of active merges instead of cumulative The cumulative number of active merges could be red herring as it its value depends on the number of vmstorages. For example, vmstorage could be added or removed and this will affect the panel. Or, each vmstorage could start a merging process (i.e. for downsampling) and visiually it could look like a massive change. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `035de57e5e`)	2024-04-24 17:08:15 +02:00
hagen1778	fcff6cc12a	docs/changelog: mention icnreased CPU usage in 1.100.1 release for ENT distributions Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `22cfab1ea2`)	2024-04-24 17:08:13 +02:00
hagen1778	57c841669c	app/vmagent: mention corner case with dangling queues and identical URLs See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6140 We don't cover this corner case as it has low chance for reproduction. Precisely, the requirements are following: 1. vmagent need to be configured with multiple identical `remoteWrite.url` flags; 2. At least one of the persistent queues need to be non-empty, which already signalizes about issues with setup; 3. vmagent need to be restarted with removing of one of `remoteWrite.url` flags. We do not document this case in vmagent.md as it seems to be a rare corner case and its explanation will require too much of explanation and confuse users. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4251292708`)	2024-04-23 14:52:35 +02:00
Github Actions	f2897fb859	Automatic update operator docs from VictoriaMetrics/operator@e343a8e (#6168 ) (cherry picked from commit `301bd387d4`)	2024-04-23 14:52:33 +02:00
Andrii Chubatiuk	29f9b8ef5d	deployment/docker: allow cross-platform building on arm64 platform (#6158 ) Added x86_64 libraries to allow building cross-platform images on arm64 (cherry picked from commit `7958f38864`)	2024-04-23 14:52:30 +02:00
Denys Holius	3c8aba3029	deployment/docker/victorialogs/fluentbit-docker/docker-compose.yml: update fluentbit version from v2.1.4 to v3.0.2 (#6120 ) see also https://fluentbit.io/announcements/v3.0.0/ (cherry picked from commit `bde4693a90`)	2024-04-23 14:52:28 +02:00
Roman Khavronenko	2566e19306	app/vmalert: fix links with anchors in vmalert's UI (#6146 ) Starting from v1.99.0 vmalert could ignore anchors pointing to specific rule groups if `search` param was present in URL. This change makes anchors compatible with `search` param in UI. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5f487c7090`)	2024-04-22 15:05:23 +02:00
Github Actions	d1ee860d46	Automatic update operator docs from VictoriaMetrics/operator@7e2ba6f (#6165 ) (cherry picked from commit `6dbb4c1671`)	2024-04-22 15:05:17 +02:00
Denys Holius	cad2cb2b56	docs: removed code-style highlighting for commanad-line flags of VM components (#6147 ) Using `sh` or `console` formatting doesn't do word-breaking on render. This makes flags description harder to read, as users need to scroll the web page horizontally. Removing the formatting renders the description with normal word-breaking. (cherry picked from commit `9bedbcfa2f`)	2024-04-22 15:04:49 +02:00
hagen1778	342290275e	app/streamaggr: follow-up after `c0e4ccb7b5` * rm vmagent mentions from vminsert flags * improve documentation wording, add links to related sections * mention `ignore_first_intervals` in the stream aggr options * update flags description * add basic test for config parsing validation Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `bae3874e6a`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-22 14:39:23 +02:00
Andrii Chubatiuk	131367fb59	lib/streamaggr: add option to ignore first N aggregation intervals (#6137 ) Stream aggregation may yield inaccurate results if it processes incomplete data. This issue can arise when data is sourced from clients that maintain a queue of unsent data, such as Prometheus or vmagent. If the queue isn't fully cleared within the aggregation interval, only a portion of the time series may be included in that period, leading to distorted calculations. To mitigate this we add an option to ignore first N aggregation intervals. It is expected, that client queues will be cleared during the time while aggregation ignores first N intervals and all subsequent aggregations will be correct. (cherry picked from commit `c0e4ccb7b5`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-22 14:34:36 +02:00
Aliaksandr Valialkin	ff122798b1	docs/MetricsQL.md: add links to `raw samples` chapter at https://docs.victoriametrics.com/keyconcepts/	2024-04-21 00:22:03 +02:00
Aliaksandr Valialkin	946757fa5e	docs/CONTRIBUTING.md: small re-wording	2024-04-21 00:08:02 +02:00
Aliaksandr Valialkin	c254a889f7	docs: remove duplicate information regarding contribution guidelines to VictoriaMetrics Substitute duplicate information with the link to https://docs.victoriametrics.com/contributing/ across all the docs.	2024-04-21 00:01:33 +02:00
Aliaksandr Valialkin	aa496d8827	docs/CONTRIBUTING.md: more formatting fixes and clarifications	2024-04-20 23:49:13 +02:00
Aliaksandr Valialkin	c87e383634	.github: move pull_request_template.md from .github/PULL_REQUEST_TEMPLATE/ to .github/ I couldn't figure out how to show the provided pull request template to users according to https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository , I let's try 'global pull request for the whole repository' approach.	2024-04-20 23:43:06 +02:00
Aliaksandr Valialkin	59648c880a	docs/CONTRIBUTING.md: small corrections after `a402847eb6`	2024-04-20 23:22:04 +02:00
Aliaksandr Valialkin	b16dbab137	Move CONTRIBUTING.md to docs/ It is better from visibility PoV if the CONTRIBUTING.md file is visible at https://docs.victoriametrics.com/contributing/ . This page can be indexed by search engines and searched by our users later. It is also easier to provide a link to this page now comparing to the old link, which is much harder to remember: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/CONTRIBUTING.md While at it, syncrhonize docs/CONTRIBUTING.md with .github/PULL_REQUEST_TEMPLATE/pull_request_template.md , so they do not contain duplicate information, which can be outdated over time. Now all the relevant information is located at docs/CONTRIBUTING.md, while .github/PULL_REQUEST_TEMPLATE/pull_request_template.md refers this doc. This is a follow-up for `c006db1798` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6040	2024-04-20 23:12:53 +02:00
Aliaksandr Valialkin	4318f34644	lib/protoparser: substitute hybrid channel-based pools with plain sync.Pool Using plain sync.Pool simplifies the code without increasing memory usage and CPU usage. So it is better to use plain sync.Pool from readability and maintainability PoV. This is a follow-up for `8942f290eb`	2024-04-20 22:02:39 +02:00
Aliaksandr Valialkin	c146b24196	app/vminsert: replace hybrid sync.Pool+channel-based pool scheme for poolCtx with plain sync.Pool This simplifies the code, while doesn't increase memory usage under low and high data ingestion rate. This is a follow-up for `1decbcf6eb`	2024-04-20 21:46:11 +02:00
Aliaksandr Valialkin	71d0020c2f	app/vminsert/influx: replace hybrid channel-based pool+sync.Pool with plain sync.Pool for pushCtx The memory usage for plain sync.Pool doesn't increase comparing to the memory usage for the hybrid scheme, so it is better to use plain sync.Pool in order to simplify the code and make it more readable and maintainable. This is a follow-up for `c22da2f917`	2024-04-20 21:41:18 +02:00
Aliaksandr Valialkin	a249ab96b4	app/vmagent/influx: replace hybrid channel-based pool + sync.Pool with plain sync.Pool for pushCtx Data ingestion benchmark doesn't show memory usage difference between two approaches, so let's use simpler approach in order to improve code readability and maintainability. This is a follow-up for `77c597738c`	2024-04-20 21:38:25 +02:00
Aliaksandr Valialkin	d6da68ee90	app/vmagent/common: use plain sync.Pool instead of a mix of sync.Pool with channel-based pool for PushCtx This scheme was used for reducing memory usage when vmagent runs on a machine with big number of CPU cores and the ingestion rate isn't too big. The scheme with channel-based pool could reduce memory usage, since it minimizes the number of PushCtx structs in the pool in this case. Performance tests didn't reveal significant difference in memory usage under both low and high ingestion rate between plain sync.Pool and the current hybrid scheme, so replace the scheme with plain sync.Pool in order to simplify the code.	2024-04-20 21:31:14 +02:00
Aliaksandr Valialkin	9004bc098e	all: use clear() built-in Go function for clearing []prompbmarshal.TimeSeries and []prompbmarshal.Label slices This makes the code a bit clear.	2024-04-20 21:00:24 +02:00
Aliaksandr Valialkin	498fe1cfa5	app/vminsert/common: remove obsolete optimization for reducing memory usage for InsertCtx pool This optimization is no longer needed according to benchmarks with ingestion rate. This simplifies the code a bit.	2024-04-20 20:51:38 +02:00
Bartosz Fenski	b36661539f	typo (#6159 ) s/infromation/information/	2024-04-20 20:49:16 +02:00
Github Actions	ac7746d7fa	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@5b8a0ba (#6156 )	2024-04-20 20:48:44 +02:00
Aliaksandr Valialkin	fba3c10ed1	app/vmselect/promql: add support for matching against multiple numeric constants via `q == (c1,...,cN)` and `q != (c1,...,cN)` syntax	2024-04-19 17:57:09 +02:00
Aliaksandr Valialkin	5504b6f2bd	docs/CHANGELOG.md: document v1.97.4 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.4	2024-04-19 13:10:37 +02:00
Aliaksandr Valialkin	a40a900b43	docs/CHANGELOG.md: document v1.93.14 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14	2024-04-19 12:52:38 +02:00
Aliaksandr Valialkin	a674029111	docs/LTS-releases.md: change the latest v1.93.x LTS release from v1.93.13 to v1.93.14 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.14	2024-04-19 12:52:36 +02:00
Aliaksandr Valialkin	5b29be1f4d	app/vmagent/remotewrite: add support for replication additionally to sharding when both -remoteWrite.shardByURL and -remoteWrite.shardByURLReplicas=RF command-line flags are set This allows setting up data replication among failure domains if the replication factor is smaller than the number of failure domains. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054 See https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages	2024-04-19 11:37:04 +02:00
Github Actions	0fde6f4b8f	Automatic update operator docs from VictoriaMetrics/operator@3d7bcba (#6149 )	2024-04-19 11:36:38 +02:00
Github Actions	66f3f90f74	Automatic update operator docs from VictoriaMetrics/operator@781dbba (#6143 )	2024-04-19 11:35:58 +02:00
Github Actions	a0b1c95787	Automatic update operator docs from VictoriaMetrics/operator@731de81 (#6139 ) Co-authored-by: Alexander Marshalov <_@marshalov.org>	2024-04-19 11:35:21 +02:00
Github Actions	3cb3a9c950	Automatic update operator docs from VictoriaMetrics/operator@7f6c6d8 (#6138 )	2024-04-19 11:34:50 +02:00
Hui Wang	e0d47ab6af	vmalert: avoid blocking APIs when alerting rule uses template functio… (#6129 ) * vmalert: avoid blocking APIs when alerting rule uses template function `query` * app/vmalert: small refactoring * simplify labels and templates expanding * simplify `newAlert` interface * fix `TestGroupStart` which mistakenly skipped annotations and response labels check Signed-off-by: hagen1778 <roman@victoriametrics.com> * reduce alerts lock time when restore --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-04-19 11:30:40 +02:00
Roman Khavronenko	95b0f82c9b	app/vmalert: make `TestGroupStart` more reliable (#6130 ) There was a sleep statement in the test, waiting for Group to perform a couple of evaluation. But looks like it worked unreliable for some CI tests like the one below https://github.com/VictoriaMetrics/VictoriaMetrics/actions/runs/8718213844/job/23915007958?pr=6115 This commit changes the sleep statement on a function that waits for a specific number of evaluations. It should make this test faster in general case, and more reliable for slow environemnts.	2024-04-19 11:28:30 +02:00
Aliaksandr Valialkin	f3967737b0	docs/Cluster-VictoriaMetrics.md: typo fix after `dadeb6620a`: folling -> following	2024-04-19 00:24:52 +02:00
Aliaksandr Valialkin	90794e84bc	app/vmselect/netstorage: add support for the ability to set cross-group replication factor at vmselect The cross-group replication factor can be set via `-globalReplicationFactor` command-line flag at vmselect. In this case vmselect continues returning full responses if up to globalReplicationFactor-1 groups are unavailable. See https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6054	2024-04-19 00:16:04 +02:00
Aliaksandr Valialkin	e138f4827e	lib/storage: search for all the values for the given label before applying filters and limits It is incorrect applying the limit on the number of values to search without applying filters, since the returned subset of label values may miss the label values matching the given filters. This is a follow-up for `66630c7960`	2024-04-18 20:40:19 +02:00
Aliaksandr Valialkin	b523893a67	app/{vlselect,vmselect}: run `make vmui-update vmui-logs-update`	2024-04-18 17:37:28 +02:00
Aliaksandr Valialkin	e2bc49534b	all: replace old https://docs.victoriametrics.com/PerTenantStatistic.html url with the new one - https://docs.victoriametrics.com/pertenantstatistic/	2024-04-18 03:36:23 +02:00
Aliaksandr Valialkin	4f1eb03441	all: replace old https://docs.victoriametrics.com/Quick-Start.html url with the new one - https://docs.victoriametrics.com/quick-start/	2024-04-18 03:33:01 +02:00
Aliaksandr Valialkin	b1477aa2be	all: replace old https://docs.victoriametrics.com/scrape_config_examples.html url with the new one - https://docs.victoriametrics.com/scrape_config_examples/	2024-04-18 03:29:54 +02:00
Aliaksandr Valialkin	87641fa7e7	all: replace old https://docs.victoriametrics.com/Troubleshooting.html url with the new one - https://docs.victoriametrics.com/troubleshooting/	2024-04-18 03:27:18 +02:00
Aliaksandr Valialkin	0c87cbe338	all: replace old https://docs.victoriametrics.com/relabeling.html url with the new one - https://docs.victoriametrics.com/relabeling/	2024-04-18 03:23:15 +02:00
Aliaksandr Valialkin	10f3540bc6	all: replace old https://docs.victoriametrics.com/Articles.html url with the new one - https://docs.victoriametrics.com/articles/	2024-04-18 03:20:13 +02:00
Aliaksandr Valialkin	987c4a2da5	all: replace old https://docs.victoriametrics.com/CaseStudies.html url with the new one - https://docs.victoriametrics.com/casestudies/	2024-04-18 03:16:54 +02:00
Aliaksandr Valialkin	b933001d2d	all: replace old https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/single-server-victoriametrics/	2024-04-18 03:11:49 +02:00
Aliaksandr Valialkin	b0129fab30	all: replace old https://docs.victoriametrics.com/url-examples.html url with the new one - https://docs.victoriametrics.com/url-examples/	2024-04-18 03:02:11 +02:00
Aliaksandr Valialkin	a21d1fcf57	all: replace old https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/cluster-victoriametrics/	2024-04-18 02:56:28 +02:00
Aliaksandr Valialkin	819569beda	all: replace old https://docs.victoriametrics.com/FAQ.html url with the new one - https://docs.victoriametrics.com/faq/	2024-04-18 02:46:30 +02:00
Aliaksandr Valialkin	ed5744f072	all: replace old https://docs.victoriametrics.com/CHANGELOG_YYYY.html url with the new one - https://docs.victoriametrics.com/changelog_YYYY/	2024-04-18 02:42:16 +02:00
Aliaksandr Valialkin	9fa1e5d843	all: replace old https://docs.victoriametrics.com/CHANGELOG.html url with the new one - https://docs.victoriametrics.com/changelog/	2024-04-18 02:39:09 +02:00
Aliaksandr Valialkin	baf5c8d6d0	all: replace old https://docs.victoriametrics.com/keyConcepts.html url with the new one - https://docs.victoriametrics.com/keyconcepts/	2024-04-18 02:34:09 +02:00
Aliaksandr Valialkin	513e69c55e	all: replace old https://docs.victoriametrics.com/sd_configs.html url with the new one - https://docs.victoriametrics.com/sd_configs/	2024-04-18 02:28:26 +02:00
Aliaksandr Valialkin	728bcf0585	all: replace old https://docs.victoriametrics.com/stream-aggregation.html url with the new one - https://docs.victoriametrics.com/stream-aggregation/	2024-04-18 02:20:00 +02:00
Aliaksandr Valialkin	64938732e3	all: replace old https://docs.victoriametrics.com/MetricsQL.html url with the new one - https://docs.victoriametrics.com/metricsql/	2024-04-18 02:15:33 +02:00
Aliaksandr Valialkin	2ec18a6634	all: replace old https://docs.victoriametrics.com/vmgateway.html url with the new one - https://docs.victoriametrics.com/vmgateway/	2024-04-18 02:08:53 +02:00
Aliaksandr Valialkin	a501e237d2	all: replace old https://docs.victoriametrics.com/vmbackupmanager.html url with the new one - https://docs.victoriametrics.com/vmbackupmanager/	2024-04-18 02:04:39 +02:00
Aliaksandr Valialkin	a85602bb2d	all: replace old https://docs.victoriametrics.com/vmrestore.html url with the new one - https://docs.victoriametrics.com/vmrestore/	2024-04-18 02:01:18 +02:00
Aliaksandr Valialkin	6fe21520ce	all: replace old https://docs.victoriametrics.com/vmbackup.html url with the new one - https://docs.victoriametrics.com/vmbackup/	2024-04-18 01:58:00 +02:00
Aliaksandr Valialkin	c7866f3e19	all: replace old https://docs.victoriametrics.com/vmctl.html url with the new one - https://docs.victoriametrics.com/vmctl/	2024-04-18 01:54:16 +02:00
Aliaksandr Valialkin	728aceebfe	all: replace old https://docs.victoriametrics.com/vmauth.html url with the new one - https://docs.victoriametrics.com/vmauth/	2024-04-18 01:50:16 +02:00
Aliaksandr Valialkin	a99005eff6	all: replace old https://docs.victoriametrics.com/vmalert.html url with the new one - https://docs.victoriametrics.com/vmalert/	2024-04-18 01:44:54 +02:00
Aliaksandr Valialkin	c0457ac11a	all: replace remaining https://docs.victoriametrics.com/vmagent.html urls with the new one - https://docs.victoriametrics.com/vmagent/	2024-04-18 01:36:20 +02:00
Aliaksandr Valialkin	0211a04a52	all: replace the outdated url https://docs.victoriametrics.com/vmagent.html with the new one - https://docs.victoriametrics.com/vmagent/	2024-04-18 01:32:57 +02:00
Aliaksandr Valialkin	e615bcced7	lib/storage: improve performance for /api/v1/label/labelName/values when match[] contains only a single filter on labelName This speeds up auto-suggestion for metric names in VMUI and Grafana, which use the following query in this case: /api/v1/label/__name__/values?match[]={__name__=~".some_value."} When the user types `some_value` in the query input field.	2024-04-18 01:18:03 +02:00
Aliaksandr Valialkin	164032cd9b	lib/httpserver: add support for automatic issuing of TLS certificates via Lets Encrypt service Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949	2024-04-17 23:53:51 +02:00
Aliaksandr Valialkin	30c96ba8d7	app/{vminsert,vmselect}: support for `srv+addr` scheme for specifying DNS SRV addresses at -storageNode flag The new scheme is consistent with SRV urls introduced at `b426d10847` and `dc326f70b4` Deprecte the old scheme: `dns+srv:addr` by removing it from the docs. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053	2024-04-17 23:15:05 +02:00
Aliaksandr Valialkin	104cb56a80	vendor: run `make vendor-update`	2024-04-17 22:55:08 +02:00
Aliaksandr Valialkin	2177675b34	lib/netutil: move creation of GetCertificate callback into a separate function This improves code readability a bit	2024-04-17 22:11:10 +02:00
Aliaksandr Valialkin	9f33b3760d	docs/vmagent.md: typo fixes after `dc326f70b4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053	2024-04-17 21:13:01 +02:00
Aliaksandr Valialkin	bb04943a2a	vendor: run `make vendor-update`	2024-04-17 21:01:21 +02:00
Aliaksandr Valialkin	284d99e269	app/vmagent: support for DNS SRV urls at -remoteWrite.url, scrape target urls and service discovery urls Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053	2024-04-17 20:56:23 +02:00
Aliaksandr Valialkin	e627810146	app/vmauth: add support for configuring backends via DNS SRV urls	2024-04-17 20:56:21 +02:00
Aliaksandr Valialkin	6cc80ab621	app/vmauth: add support for client TLS sertificates for backend requests over https While at it, also add support for TLS ServerName for backend requests over https	2024-04-17 17:13:12 +02:00
Aliaksandr Valialkin	93b606cefb	app/vmauth: use lib/promauth for creating backend roundtripper This simplifies further maintenance and opens doors for additional config options supported by lib/promauth. For example, an ability to specify client TLS certificates.	2024-04-17 17:13:11 +02:00
Aliaksandr Valialkin	eb870d9de3	docs/vmauth.md: add Authorization and Routing chapters	2024-04-17 16:17:21 +02:00
Aliaksandr Valialkin	43cd420b84	README.md: sync with docs/Cluster-VictoriaMetrics.md after `691181867e`	2024-04-17 15:03:19 +02:00
Aliaksandr Valialkin	6cdf01297e	app/vmauth: follow-up for `b155b20de4` - Use exact matching by default for the query arg value provided via arg=value syntax at src_query_args. Regex matching can be enabled by using =~ instead of = . For example, arg=~regex. This ensures that the exact matching works as expected without the need to escape special regex chars. - Add helper functions for creating QueryArg, Header and Regex structs in tests. This improves maintainability of the tests. - Remove url.QueryUnescape() call on the url in TestCreateTargetURLSuccess(), since this is bogus approach. The url.QueryUnescape() must be applied to individual query args, and it mustn't be applied to the whole url, since in this case it may perform invalid unescaping in the context of the url, or make the resulting url invalid. While at it, properly marshal all the fields inside UserInfo config to yaml in tests. Previously Header and QueryArg structs were improperly marshaled because the custom MarshalYAML is called only on pointers to Header and QueryArg structs. This improves test coverage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6115	2024-04-17 14:53:20 +02:00
Aliaksandr Valialkin	54c9f9c5ea	docs/Cluster-VictoriaMetrics.md: clarify per-node workload increase when one of vmstorage node is unavailable in the cluster The docs update is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6099#issuecomment-2060856417	2024-04-17 14:52:49 +02:00
Yury Molodov	e4d6ded0f5	vmui: preserve Select value if it matches list on blur (#6101 )	2024-04-17 14:52:30 +02:00
Roman Khavronenko	256c561005	app/vmauth: support regex matching in `src_query_args` (#6115 ) Support regex matching when routing incoming requests based on HTTP query args via `src_query_args` option at `url_map`. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-17 14:52:02 +02:00
Aliaksandr Valialkin	c6f0a37125	docs/Cluster-VictoriaMetrics.md: remove incorrect information about the increase of workload on the remaining vmstorage nodes when one vmstorage node is unavailable If one out of 5 vmstorage nodes is unavailable, then the remaining 4 vmstorage nodes will recieve 1/5=20% increase of workload, not 5%. If one out of 10 vmstorage nodes is unavailable, then the remaining 9 vmstorage nodes will receive 1/10=10% increase of workload, not 1%. This is a follow-up for `458338afa5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6099	2024-04-17 14:49:59 +02:00
Zakhar Bessarab	3f618c0485	app/vmauth: do not increment backend_errors when hitting concurrency limit (#6078 ) * app/vmauth: do not increment backend_errors when hitting concurrency limit Previously, both "vmauth_concurrent_requests_limit_reached_total" and "vmauth_user_request_backend_errors_total" were incremented. This was based on the assumption that if concurrency limit is hit the backend must be failing to handle the request thus meaning an error. This assumption does not work in case the endpoint can be overloaded by the misbehaving client sending too many requests within the timeframe. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565 Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-04-17 12:34:33 +02:00
Github Actions	60a447bb76	Automatic update operator docs from VictoriaMetrics/operator@99fbc98 (#6122 )	2024-04-17 12:15:39 +02:00
Aliaksandr Valialkin	2347699126	docs/CHANGELOG.md: move the description for the bugfix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110 from v1.100.1 to the `tip` section The bugfix isn't included in v1.100.1 release. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6110 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6111 This is a follow-up for `7308bad777`	2024-04-17 12:13:01 +02:00
Aliaksandr Valialkin	a9ec77fee5	lib/promscrape/discovery/consul: typo fix in the comment: enteprise -> enterprise	2024-04-17 12:08:01 +02:00
Aliaksandr Valialkin	5320cc3198	lib/{mergeset,storage}: log deleting directories inside partitions if they are missing in parts.json This should improve debuggability of unexpected deletion of directories inside partitions. While at it, log the proper path to parts.json when the directory for big part is missing in the partition. parts.json is located inside directory with small parts, and there is no parts.json file inside directory with big parts.	2024-04-17 12:00:10 +02:00
Aliaksandr Valialkin	9b2a5c8844	lib/storage: improve comments inside functions responsible for creating indexes for newly registered time series	2024-04-17 11:30:51 +02:00
Zakhar Bessarab	691181867e	docs/cluster: update cluster resizing info (#6099 ) * docs/cluster: update cluster resizing info - add example of resources distribution - add info about how to handle uneven disk usage after adding a new storage node Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `458338afa5`)	2024-04-17 09:40:36 +02:00
Github Actions	a8444cb54b	Automatic update operator docs from VictoriaMetrics/operator@13f6dac (#6119 ) (cherry picked from commit `aaa18e565d`)	2024-04-17 09:40:34 +02:00
hagen1778	3bddb10af8	docs: mention HTTP sink configuration example for Vector Follow-up `16eeb4e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4f55aa29db`)	2024-04-17 09:40:32 +02:00
hagen1778	1660428e00	deployment/vector: add example for JSON stream config Follow-up `16eeb4eb33` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9064602d00`)	2024-04-17 09:40:30 +02:00
Devin Buhl	a09da7a42c	victorialogs: mention vector supports http/json stream (#6114 ) https://github.com/vectordotdev/vector/issues/18883#issuecomment-1771424716 (cherry picked from commit `16eeb4eb33`)	2024-04-17 09:40:27 +02:00
guangwu	c150924b81	app/vmctl: properly close file descriptor in `verify-block` action (#6106 ) (cherry picked from commit `9dd5db2b77`)	2024-04-17 09:40:23 +02:00
Vadim Rutkovsky	59fc201aee	dashboards: fix typo in VictoriaLogs panel (#6102 ) Comprasion -> compression (cherry picked from commit `66c5fc3243`)	2024-04-16 09:58:33 +02:00
yudrywet	d89229c838	chore: fix some typos in comments (#6103 ) Signed-off-by: yudrywet <yudeyao@yeah.net> (cherry picked from commit `43835704b7`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-16 09:58:30 +02:00
Alexander Marshalov	c61eb6d620	vmalert: support any status code from the range 200-299 from alertmanager as successful (#6111 ) * any status code from the range 200-299 from alertmanager to vmalert is not considered an error from now on (#6110) * add changelog (cherry picked from commit `7308bad777`)	2024-04-16 09:58:09 +02:00
Github Actions	20f16aed1a	Automatic update operator docs from VictoriaMetrics/operator@cf48a99 (#6113 ) (cherry picked from commit `7db8ba41e7`)	2024-04-16 09:58:07 +02:00
Dmytro Kozlov	097cc8a2a8	docs: fix typo in the curl command (#6109 ) (cherry picked from commit `7b20de4674`)	2024-04-16 09:58:05 +02:00
Yury Molodov	f20cd59f77	vmui/vmanomaly: integrate vmanomaly query_server (#6017 ) * vmui: fix parsing of fractional values * vmui/vmanomaly: update display logic to align with vmanomaly /query_range API * vmui/vmanomaly: rename flag anomalyView to isAnomalyView (cherry picked from commit `f06f55edb6`)	2024-04-16 09:58:03 +02:00
Dima Lazerka	ea4d0a1423	VMUI: Update builder Node version (#5908 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `22497c2c98`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-16 09:58:00 +02:00
Github Actions	4859f5eeed	Automatic update operator docs from VictoriaMetrics/operator@73a1996 (#6100 ) (cherry picked from commit `cba2f6dce1`)	2024-04-16 09:57:44 +02:00
hagen1778	b81a66ceea	docs: add 1.100.1 release date Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e39a1a98f5`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-16 09:57:41 +02:00
Hui Wang	9c0da1ef3d	deployment: bump victoriametrics components to v1.100.1 (#6096 )	2024-04-12 12:19:21 +02:00
Aliaksandr Valialkin	4179ebdce7	deployment: upgrade VictoriaLogs from v0.5.1-victorialogs to v0.5.2-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.2-victorialogs	2024-04-11 09:53:36 +02:00
Aliaksandr Valialkin	c8a46b0ad2	vendor: run `make vendor-update`	2024-04-11 09:46:29 +02:00
Aliaksandr Valialkin	14448966d9	docs/VictoriaLogs/CHANGELOG.md: cut v0.5.2-victorialogs release	2024-04-11 09:42:22 +02:00
Aliaksandr Valialkin	67af029b7f	docs/VictoriaLogs/CHANGELOG.md: document the bugfix at `2205de2391` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6042	2024-04-11 09:42:20 +02:00
Aliaksandr Valialkin	8cb768a7a5	docs/CHANGELOG.md: cut v1.100.1 release	2024-04-11 09:35:27 +02:00
Aliaksandr Valialkin	14a6dee206	docs/CHANGELOG.md: mention that the bug with incorrect registration of new entries in the IndexDB has been introduced in v1.99.0	2024-04-11 09:35:26 +02:00
Zakhar Bessarab	da8bb106aa	lib/mergeset: fix flushing incorrect set of inmemoryBlocks (#6089 ) Follow-up for `bace9a2501` Related: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6069 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-11 09:26:48 +02:00
Github Actions	26ed8d0571	Automatic update operator docs from VictoriaMetrics/operator@17082f0 (#6090 )	2024-04-11 09:15:21 +02:00
Zakhar Bessarab	956ff263fc	docs/changelog: add entry for bd398974334678a29972475b0566e0bd2434d197 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-04-11 09:12:12 +02:00
飞	7c4c6f7ee6	docs: fix a typo in docs/operator/resources/vmalert.md (#6088 )	2024-04-11 09:09:26 +02:00
Denys Holius	4aaef26e3c	fixed EXPOSE port for VictoriaLogs' Dockerfiles (#6082 )	2024-04-11 09:08:44 +02:00
Artem Navoiev	deba9a66ff	github actions: sync doc action, do not build search index, just copy content Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-04-08 15:40:36 +02:00
wanshuangcheng	52a4ae0b28	chore: fix function names in comment (#6076 ) Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>	2024-04-08 15:38:51 +02:00
Aliaksandr Valialkin	e46150c3bf	docs/CHANGELOG.md: remove description of the reverted change at `c79bf3925c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985	2024-04-04 16:56:04 +03:00
Aliaksandr Valialkin	b8d5fe0584	deployment/docker: update VictoriaLogs tag from v0.5.0-victorialogs to v0.5.1-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.1-victorialogs	2024-04-04 16:52:06 +03:00
Aliaksandr Valialkin	a549fc2941	docs/VictoriaLogs/CHANGELOG.md: cut v0.5.1-victorialogs release	2024-04-04 16:41:38 +03:00
Aliaksandr Valialkin	c55afb9a18	docs/VictoriaLogs/CHANGELOG.md: document the fix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5927 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920 This is a follow-up for `46fd0ed693`	2024-04-04 16:39:39 +03:00
Aliaksandr Valialkin	fe28388c5d	deployment: update VictoriaMetrics docer image from v1.99.0 to v1.100.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.100.0	2024-04-04 15:29:12 +03:00
Aliaksandr Valialkin	18e521dbff	docs: change old url from https://docs.victoriametrics.com/enterprise.html to new url https://docs.victoriametrics.com/enterprise/	2024-04-04 15:22:38 +03:00
Aliaksandr Valialkin	68a9bcaa1e	docs/Single-server-VictoriaMetrics.md: remove misleading `filters` word from the text, which recommends evaluating downsampling in VictoriaMetrics enterprise	2024-04-04 14:58:45 +03:00
Aliaksandr Valialkin	6cf8d2144e	docs: clarify that downsampling drops all the samples except the last one on every downsampling interval	2024-04-04 14:50:24 +03:00
Aliaksandr Valialkin	34687cfec4	docs/CHANGELOG.md: typo fix: remove duplicate `samples` word	2024-04-04 14:46:21 +03:00
Aliaksandr Valialkin	6332a1c447	docs/CHANGELOG.md: advise upgrading from v1.99.0 to v1.100.0 because of the issue at v1.99.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959	2024-04-04 14:41:31 +03:00
Aliaksandr Valialkin	283f4b96f6	docs/CHANGELOG.md: move custom backup interval feature to `tip`, since it wasnt included in v1.100.0 release Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966 This is a follow-up for baa869742208f311d84e800ea527a7f07eb1ca18	2024-04-04 12:33:49 +03:00
Zakhar Bessarab	27f5eb38bb	Vmbackupmanager: add support of custom backup interval (#742 ) * app/vmbackupmanager: add support of defining custom backup interval See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: update vmbackupmanager docs Update docs after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-04-04 12:33:48 +03:00
Aliaksandr Valialkin	530d7b40bd	docs/CHANGELOG.md: add release date for v1.100.0	2024-04-04 12:28:31 +03:00
Aliaksandr Valialkin	ddb1b5b70f	docs/CHANGELOG.md: cut v1.100.0	2024-04-04 03:47:32 +03:00
Aliaksandr Valialkin	222e8b5a7b	lib/streamaggr: update the minimum allowed timestamp for incoming samples before flushing the samples to the storage This should prevent from dropping samples with old timestamps during long flushes. This is a follow-up for `1cedaf61cb`	2024-04-04 02:26:08 +03:00
Aliaksandr Valialkin	d61f6c89a1	app/{vmagent,vminsert}: accept Prometheus remote write protocol requests at /prometheus/api/v1/push additionally to /api/v1/push This is a follow-up for `7ccdb57ea4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5990	2024-04-04 02:17:10 +03:00
Eugene Ma	7459a197c6	add "/api/v1/push" to request handler (#5990 ) Co-authored-by: Eugene Ma <eugene.ma@airbnb.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 02:16:27 +03:00
Aliaksandr Valialkin	fc4a5cc022	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update` after the recent changes at app/vmui	2024-04-04 02:09:39 +03:00
nemobis	fdfdc9f8f1	Add note about final deduplication space needs (#5996 ) Addresses #5975 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 02:09:12 +03:00
Zakhar Bessarab	760abf6186	[docs][github] Update contributing information (#6040 ) * add pull request template Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * update text Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * update text Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * Update .github/PULL_REQUEST_TEMPLATE/pull_request_template.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update .github/PULL_REQUEST_TEMPLATE/pull_request_template.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * update messaging add example Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * docs/contributing: add info about mandatory requirements before sending a PR Added the following info: - commits signing / sign-off - how to run tests / linting with make commands Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2024-04-04 02:02:11 +03:00
dependabot[bot]	cfb8b19357	build(deps-dev): bump express in /app/vmui/packages/vmui (#6038 ) Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](https://github.com/expressjs/express/compare/4.18.2...4.19.2) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 01:57:16 +03:00
dependabot[bot]	86ff78be39	build(deps-dev): bump webpack-dev-middleware in /app/vmui/packages/vmui (#6011 ) Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.3 to 5.3.4. - [Release notes](https://github.com/webpack/webpack-dev-middleware/releases) - [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md) - [Commits](https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4) --- updated-dependencies: - dependency-name: webpack-dev-middleware dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yury Molodov <yurymolodov@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 01:55:56 +03:00
dependabot[bot]	e07384055d	build(deps-dev): bump follow-redirects in /app/vmui/packages/vmui (#5978 ) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.5...v1.15.6) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yury Molodov <yurymolodov@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 01:54:55 +03:00
Yury Molodov	2d8777ac1e	vmui: fix step update on input blur in Firefox/Safari (#6034 ) * vmui: fix step value application on input blur * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 01:53:15 +03:00
Yury Molodov	da642b56a1	vmui: fix trigger auto-suggestion (#6033 ) * vmui: fix ui freeze on query paste #5923 * vmui: fix auto-suggestion trigger issue after whitespace char #5866 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-04 01:49:17 +03:00
Aliaksandr Valialkin	7abdba4573	app/vmui/Dockerfile-web: update alpine docker image from 3.19.0 to 3.19.1 This is a follow-up for `fcc8b14f86`	2024-04-04 01:45:40 +03:00
Aliaksandr Valialkin	ecf3dfde9a	deployment: update Go builder from 1.22.1 to 1.22.2 See https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved	2024-04-04 01:44:58 +03:00
Aliaksandr Valialkin	2f07043e74	vendor: run `make vendor-update`	2024-04-04 01:42:59 +03:00
Aliaksandr Valialkin	ecd782c75e	app/vmagent: follow-up for `b3b29ba6ac` - Automatically reload changed TLS root CA pointed by -remoteWrite.tlsCAFile command-line flag - Automatically reload changed TLS root CA configured via oauth2.tsl_config.ca_file option at -promscrape.config - Document the change as a feature instead of a bug at docs/CHANGELOG.md - Simplify the code at lib/promauth, which is responsible for reloading changed TLS root CA files. - Simplify the usage of lib/promauth.Config.NewRoundTripper() - now it accepts the base http.Transport instead of a callback, which can change the internal http.Transport. - Reuse the default tls config if lib/promauth.Config doesn't contain tls-specific configs. This should reduce memory usage a bit when tls isn't used for scraping big number of targets. - Do not re-read TLS root CA files on every processed request. Re-read them once per second. This should reduce CPU usage when scraping big number of targets over https. - Do not store cert.pem and key.pem files in TestTLSConfigWithCertificatesFilesUpdate, since they can be loaded from byte slices via crypto/tls.X509KeyPair(). - Remove obsolete comparisons of string representations for authConfig and proxyAuthConfig at areEqualScrapeConfigs(). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5725 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171	2024-04-04 01:26:38 +03:00
Aliaksandr Valialkin	1c6dbe5714	docs/CHANGELOG.md: add - in front of -logInvalidAuthTokens command-line flag in order to be consistent with command-line flag naming Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029	2024-04-04 00:42:37 +03:00
Zakhar Bessarab	bb2d029526	vmgateway: add an ability to log invalid auth tokens (#743 ) * app/vmgateway: add an ability to log invalid auth tokens This is useful for debugging to make it easier for user to find issues in token contents. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6029 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: add info about new vmgateway flag - add changelog entry - add info about logInvalidAuthTokens flag Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmgateway/filters/auth: improve reject reason visibility Explicitly return a rejection reason for request when "logInvalidAuthTokens" is enabled. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-04-04 00:42:09 +03:00
Aliaksandr Valialkin	2be627f1fe	app/vmalert: document that -rule.stripFilePath command-line flag is available only in enterprise version of vmalert	2024-04-04 00:41:35 +03:00
Zakhar Bessarab	80315e07b1	lib/promscrape/config: fix missing timeout for http client (#6063 ) Follow-up for `b3b29ba6` Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-04-04 00:40:48 +03:00
Thomas	7526f1d3a5	chore(docs): fix vmalertmanager typo (#6056 ) Fixes: #6055 Signed-off-by: Thomas Way <thomas@6f.io> Co-authored-by: Alexander Marshalov <_@marshalov.org>	2024-04-04 00:39:53 +03:00
Github Actions	ecee306e7a	Automatic update operator docs from VictoriaMetrics/operator@92cdca3 (#6052 )	2024-04-04 00:39:19 +03:00
Zakhar Bessarab	da4352fe7b	lib/{promauth,promscrape}: automatically refresh root CA certificates after changes on disk (#5725 ) * lib/{promauth,promscrape}: automatically refresh root CA certificates after changes on disk Added a custom `http.RoundTripper` implementation which checks for root CA content changes and updates `tls.Config` used by `http.RoundTripper` after detecting CA change. Client certificate changes are not tracked by this implementation since `tls.Config` already supports passing certificate dynamically by overriding `tls.Config.GetClientCertificate`. This change implements dynamic reload of root CA only for streaming client used for scraping. Blocking client (`fasthttp.HostClient`) does not support using custom transport so can't use this implementation. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5526 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promauth/config: update NewRoundTripper API Update API to allow user to update only parameters required for transport. Add warning log when reloading Root CA failed. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promauth/config: fix mutex acquire logic Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promauth/config: replace RWMutex with regular mutex to simplify the code - remove additional mutex used for getRootCABytes - require callee to use mutex - replace RWMutex with regular mutex Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promauth/config: refactor - hold the mutex lock to avoid round tripper being re-created twice - move recreation logic into separate func to simplify the code Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-04-04 00:34:43 +03:00
Aliaksandr Valialkin	2e1c8bb436	docs/CHANGELOG.md: typo fix: resonses -> responses	2024-04-03 03:20:09 +03:00
Aliaksandr Valialkin	62c1545f97	docs/CHANGELOG.md: remove duplicate line left after `1e0a079143`	2024-04-03 03:19:48 +03:00
hagen1778	6a3bf16475	docs: update vmalert troubleshooting docs * rm recommendation to keep look-behind window empty, as it is not correct * mention the change of default value for `-search.latencyOffset` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-03 03:15:11 +03:00
Aliaksandr Valialkin	bb9bb600b3	lib/protoparser/opentelemetry: follow-up after `47892b4a4c` - Rename -opentelemetry.sanitizeMetrics command-line flag to more clear -opentelemetry.usePrometheusNaming - Clarify the description of the change at docs/CHANGELOG.md - Rename promrelabel.SanitizeLabelNameParts to more clear promrelabel.SplitMetricNameToTokens - Properly split metric names at '_' char in promerlabel.SplitMetricNameToTokens. - Add tests for various edge cases for Prometheus metric names' normalization according to the code at `b865505850/pkg/translator/prometheus/normalize_name.go` - Extract the code responsible for Prometheus metric names' normalization into a separate file (santize.go) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6037 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6035	2024-04-03 03:09:52 +03:00
Aliaksandr Valialkin	a5f65756f8	docs: follow-up for `ac9c2a796f` Remove description for -search.maxExportDuration and -search.maxStatusRequestDuration command-line flags from the 'Resource usage limits' chapter, since these flags are rarely used for limiting resource usage and they are already documented in the 'List of command-line flags' chapter.	2024-04-03 03:08:27 +03:00
Aliaksandr Valialkin	5874189797	app/vmselect/graphite: follow-up for `23ab865035` - Fix docs for new functions at app/vmselect/graphite/functions.json - Properly drain series lists on errors in aggregateSeriesListsGeneric() and aggregateSeriesList() - Add links to docs for the added functions at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5809	2024-04-03 03:07:15 +03:00
Aliaksandr Valialkin	00f59d6ddf	all: fix golangci-lint(revive) warnings after `0c0ed61ce7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001	2024-04-03 03:00:45 +03:00
Aliaksandr Valialkin	1ffad3a182	lib/storage: consistently use stopCh instead of stop	2024-04-03 02:54:51 +03:00
Aliaksandr Valialkin	f1b5784432	app/vmauth: add ability to authorize via any opaque HTTP request header value This can be done via `auth_token` option at -auth.config - see https://docs.victoriametrics.com/vmauth/#auth-config	2024-04-03 02:52:39 +03:00
Aliaksandr Valialkin	2e9ee89cf1	app/vmauth: add ability to read auth tokens from multiple http request headers This is needed for VictoriaMetrics Cloud, where the same token could be passed either via Authorization or via X-Amz-Firehose-Access-Key header - see `4487dac30b (r140500722)` This is a follow-up for `4487dac30b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009	2024-04-03 02:52:20 +03:00
Artem Navoiev	f668489051	app/{vmagent/insert} fix typo in Firehose Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-04-03 02:51:57 +03:00
Aliaksandr Valialkin	456180b9d9	app/vmauth: follow-up for `bc90f4aae6` - Allow specifying only a single HTTP header for reading auth tokens via -httpAuthHeader command-line flag. This is better from security PoV, since this prevents from accidental reading of auth token from undesired HTTP header. By default the -httpAuthHeader equals to Authorization. When it is overridden, then auth token isn't read from Authorization header - it is read only from the specified header. - Document the -httpAuthHeader command-line flag at https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009	2024-04-03 02:51:28 +03:00
Aliaksandr Valialkin	faa2ba828a	app/vmagent: simplify code after `509df44d03` - Simplify the code in order to improve its maintenance - Properly pass tenant ID when processing multi-tenant opentelemetry request at vmagent Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6016	2024-04-03 02:50:46 +03:00
Fred Navruzov	28f7b2218a	docs/vmanomaly: typos fix (#6047 )	2024-04-03 02:45:35 +03:00
Fred Navruzov	d83e05a3e9	docs/vmanomaly: v1.12 updates & fixes (#6046 ) * docs/vmanomaly: v1.12.0 & link updates * add autotuned description to model section * - update refs of vmanomaly on enterprise and vmalert pages - add diagrams for model types - update self-monitoring section * - fix typos - remove .index.html from links	2024-04-03 02:44:58 +03:00
Aliaksandr Valialkin	e51190a34c	Revert "app/vmselect: make vmselect resilient to absence of cache folder (#5987 )" This reverts commit `cb23685681`. Reason for revert: the "fix" may hide programming bugs related to incorrect creation of folders before their use. This may complicate detecting and fixing such bugs in the future. There are the following fixes for the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985 : - To configure the OS to do not drop data from the system-wide temporary directory (aka /tmp). - To run VictoriaMetrics with -cacheDataPath command-line flag, which points to the directory, which cannot be removed automatically by the OS. The case when the user accidentally deletes the directory with some files created by VictoriaMetrics shouldn't be considered as expected, so VictoriaMetrics shouldn't try resolving this case automatically. It is much better from operation and debuggability PoV is to crash with the clear `directory doesn't exist` error in this case.	2024-04-03 02:44:00 +03:00
Aliaksandr Valialkin	db6d5d6349	docs/CHANGELOG.md: mention that the bug with improper use of -search.maxExportDuration instead of -search.maxLabelsAPIDuration has been introduced in v1.99.0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5992 This is a follow-up for `bc79f7196d`	2024-04-03 02:43:40 +03:00
Aliaksandr Valialkin	83652eea1d	docs/VictoriaLogs/CHANGELOG.md: improve the description of the bugfix from `43b5d8bc7a`, so it can be googled by users	2024-04-03 02:43:21 +03:00
Aliaksandr Valialkin	7edb5f77f1	app/vmagent: properly shutdown when -maxIngestionRate limit is reached The remotewrite.Stop() expects that there are no pending calls to TryPush(). This means that the ingestionRateLimiter.Register() must be unblocked inside TryPush() when calling remotewrite.Stop(). Provide remotewrite.StopIngestionRateLimiter() function for unblocking the rate limiter before calling the remotewrite.Stop(). While at it, move the rate limiter into lib/ratelimiter package, since it has two users. Also move the description of the feature to the correct place at docs/CHANGELOG.md. Also cross-reference -remoteWrite.rateLimit and -maxIngestionRate command-line flags. This is a follow-up for `02bccd1eb9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900	2024-04-03 02:41:11 +03:00
Aliaksandr Valialkin	6719415750	docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md: user proper image paths according to docs/assets/README.md	2024-04-03 02:40:51 +03:00
Aliaksandr Valialkin	7d98632c9d	docs/managed-victoriametrics: use proper names for the linked images according to docs/assets/README.md This is a follow-up for `db3709c87d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5989	2024-04-03 02:40:31 +03:00
Aliaksandr Valialkin	968c5e5dec	vendor: update github.com/VictoriaMetrics/metrics and github.com/VictoriaMetrics/metricsql to newer versions This is needed for updating broken links to MetricsQL docs: https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL -> https://docs.victoriametrics.com/metricsql/ This is a follow-up for `7e3511ffbd`	2024-04-03 02:40:08 +03:00
Aliaksandr Valialkin	eff9676508	docs/CHANGELOG.md: remove the update notes regarding converting custom HTTP header keys to canonical form Custom HTTP headers are set via net/http.Header.Set or net/http.Header.Add functions. These functions always convert header keys to canonical form. So the change at `b577413d3b` isn't visible to users of VictoriaMetrics components. There is no need in documenting this change at docs/CHANGELOG.md, since it doesn't give any useful information to users. This is a follow-up for `e6dd52b04c`	2024-04-03 02:39:08 +03:00
Zakhar Bessarab	3d15a31c6d	lib/storage: add ability to use downsampling for the given series filter (#733 ) * lib/storage: add ability to use downsampling for the given series filter Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: add information about downsampling filters Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: fix MetricsQL filter Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/downsampling: treat missing downsampling filter as a bug Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/part_header: verify correctness of downsampling filters when opening partition Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/downsampling: save only appliable rules in part metadata Filter and save only rules which are appliable to partition based on MinTimestamp of stored data. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/downsampling: update log messages for final dedup Properly specify a reason of re-running deduplication for partition. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage: consistently use MaxTimestamp to determine deduplication/downsampling rules Using MinTimestamp leads to applying downsampling to parts which are only partially covered by downsampling rule. For example, partition covers range [1000-2000]. At t=2100 and rule offset 500 data with t=2100-500 => 1600 must be downsampled. The range check against MinTimestamp evaluates to true even though partition contains range which must not be downsampled - [1600:2000]. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Follow-up - Apply the first matching downsampling period if multiple filters match the given time series. This allows fine-tuning the downsampling config for the specific needs. - Take into account downsampling filters during search queries. - Reduce the difference between community and enterprise branches. This should simplify further maintenance of these branches. - Properly parse series filters with colons inside them. - Document the feature at docs/CHANGELOG.md. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4960 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-03 02:38:37 +03:00
Aliaksandr Valialkin	ae6190f0b5	lib/storage/table.go: reduce the difference with enterprise branch	2024-04-03 02:37:05 +03:00
Aliaksandr Valialkin	b6d1d6982e	lib/storage/partition.go: reduce code difference a bit with enterprise branch	2024-04-03 02:36:49 +03:00
Nikolay	c457f7de69	lib/storage: adds metrics for downsampling (#382 ) * lib/storage: adds metrics for downsampling vm_downsampling_partitions_scheduled - shows the number of parts, that must be downsampled vm_downsampling_partitions_scheduled_size_bytes - shows total size in bytes for parts, the must be donwsampled These two metrics answer the questions - is downsampling running? how many parts scheduled for downsampling and how many of them currently downsampled? Storage space that it occupies. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612 * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-04-03 02:36:05 +03:00
Andrii Chubatiuk	0799834aaa	opentelemetry: added cmd flag to sanitize metric names (#6035 )	2024-04-03 02:31:39 +03:00
Aliaksandr Valialkin	3f79e54a51	app/vmagent/remotewrite: follow-up for `166b97b8d0` and `b6bd9a97a3` - Make the configuration more clear by accepting the list of ignored labels during sharding via a dedicated command-line flag - -remoteWrite.shardByURL.ignoreLabels. This prevents from overloading the meaning of -remoteWrite.shardByURL.labels command-line flag. - Removed superfluous memory allocation per each processed sample if sharding by remote storage is enabled. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938	2024-04-03 00:50:48 +03:00
Dmytro Kozlov	111a8e5fd0	docs: describe timeout query argument (#6020 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-04-03 00:02:00 +03:00
Hui Wang	f410520cfa	docs: fix golangci-lint check (#6036 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-04-02 23:52:50 +03:00
Hui Wang	fdb6eb1071	vmalert: fix sending alert messages (#6028 ) * vmalert: fix sending alert messages 1. fix `endsAt` field in messages that send to alertmanager, previously rule with small interval could never be triggered; 2. fix behavior of `-rule.resendDelay`, before it could prevent sending firing message when rule state is volatile. * docs: update changelog notes Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-04-02 23:51:06 +03:00
Aliaksandr Valialkin	93b389eb70	docs/MetricsQL.md: typo fix: outlier_iqr_over_time(memory_usage_bytes[1h]) triggers when memory_usage_bytes goes outside the usual value range for the last hour, not the last 24 hours This is a follow-up for `ea81f6fc36`	2024-04-02 23:48:58 +03:00
hagen1778	467c1ba617	docs: mention new guide `How to use OpenTelemetry metrics with VictoriaMetrics` in docs Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-04-02 23:47:44 +03:00
Nikolay	2ee8e462f8	docs/opentelemetry: adds opentemetry get started guide (#5861 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>	2024-04-02 23:47:06 +03:00
Aliaksandr Valialkin	b8d37ad747	lib/storage: follow-up for `76f00cea6b` Store the deadline when the metricID entries must be deleted from indexdb if metricID->metricName entry isn't found after the deadline. This should make the code more clear comparing the the previous version, where the timestamp of the first metricID->metricName lookup miss was stored in missingMetricIDs. Remove the misleading comment about the importance of the order for creating entries in the inverted index when registering new time series. The order doesn't matter, since any subset of the created entries can become visible for search before any other subset after registering in indexdb. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959	2024-04-02 23:46:21 +03:00
Github Actions	5d25fe206d	Automatic update operator docs from VictoriaMetrics/operator@c336013 (#6023 )	2024-04-02 23:46:00 +03:00
Github Actions	ee95e9d14d	Automatic update operator docs from VictoriaMetrics/operator@ac29c88 (#6022 )	2024-04-02 23:45:41 +03:00
Zakhar Bessarab	7c1ee69205	lib/storage/table: wait for merges to be completed when closing a table (#5965 ) * lib/storage/table: properly wait for force merges to be completed during shutdown Properly keep track of running background merges and wait for merges completion when closing the table. Previously, force merge was not in sync with overall storage shutdown which could lead to holding ptw ref. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: add changelog entry Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-04-02 21:25:30 +03:00
Andrii Chubatiuk	9c92cc2759	vmauth: support other auth header names besides Authorization (#6009 )	2024-04-02 18:41:00 +03:00
Andrii Chubatiuk	914b23f1e8	app/{vmagent,vminsert}: fixed firehose response (#6016 )	2024-04-02 18:03:12 +03:00
hagen1778	1c597e9dba	docs: follow-up after `623d257faf` `623d257faf` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2e843a8ed9`)	2024-03-29 14:30:25 +01:00
Jiekun	1e0a079143	app/vmalert: respect batch size limit for remote write on shutdown (#6039 ) During shutdown period of vmalert, remotewrite client retrieve all pending time series from buffer queue, compose them into 1 batch and execute remote write. This final batch may exceed the limit of -remoteWrite.maxBatchSize, and be rejected by the receiver (gateway, vmcluster or others). This changes ensures that even during shutdown vmalert won't exceed the max batch size limit for remote write destination. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6025 (cherry picked from commit `623d257faf`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-29 14:30:24 +01:00
hagen1778	36c1ca9949	app/vmagent: follow-up `166b97b8d0` * add tests for sharding function * update flags description * add changelog note `166b97b8d0` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b6bd9a97a3`)	2024-03-29 14:10:17 +01:00
Eugene Ma	744d513181	vmagent: support sharding by excluded labels (#5938 ) To horizontally scale streaming aggregation, you might want to deploy a separate hashing tier of vmagents that route to a separate aggregation tier. The hashing tier should shard by all labels except the instance-level labels, to ensure the input metrics are routed correctly to the aggregator instance responsible for those labels. For this to achieve we introduce `remoteWrite.shardByURL.inverseLabels` flag to inverse logic of `remoteWrite.shardByURL.labels` --------- Co-authored-by: Eugene Ma <eugene.ma@airbnb.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `166b97b8d0`)	2024-03-29 14:10:15 +01:00
Roman Khavronenko	548bf31dd2	app/vmselect: make vmselect resilient to absence of cache folder (#5987 ) vmselect uses a cache folder in file system for two purposes: 1. Storing rollup cache results on shutdown; 2. Storing temporary search results from vmstorage during query executions. It could happen that cache folder is deleted accidentally by user, or by OS during cleanup routines. This would cause vmselect to: 1. panic on /metrics call, because `MustGetFreeSpace` will fail; 2. return query error user, as it won't be able to store temporary search results. The changes in this commit are the following: 1. Make `MustGetFreeSpace` to try re-creating the cache folder if it is missing; 2. Make vmselect to try re-creating the cache folder if it can't persist tmp search results. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5985 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com> (cherry picked from commit `cb23685681`)	2024-03-26 15:27:32 +01:00
rbizos	ab44787a70	adding AggregateSeriesLists graphite function (#5809 ) * adding aggregate series list graphite function adding also aliases for sum diff and multiply * Adding tests for aggregateSeriesLists and aliases	2024-03-26 14:48:58 +01:00
Artem Navoiev	bf226879ec	add more redirects Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `b08cbd0400`)	2024-03-25 15:42:54 +01:00
Artem Navoiev	19a11c12d1	fix typo in kyiv city name Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `b569fa0b2c`)	2024-03-25 15:42:53 +01:00
Nikolay	60d64edd6e	app/vlselect: follow-up for `0514091948` (#6004 ) removes println lines (cherry picked from commit `43b5d8bc7a`)	2024-03-25 15:42:53 +01:00
Denys Holius	7a5f3e8ef9	Makefile: bump version of golangci-lint to the latest v1.57.1 (#6001 ) Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `0c0ed61ce7`)	2024-03-25 15:42:52 +01:00
Alexander Marshalov	0b3744effa	[vmagent] added ingestion rate limiting with new flag `-maxIngestionRate` (#5900 ) * [vmagent] added ingestion rate limiting with new flag `-maxIngestionRate`. This flag can be used to limit the number of samples ingested by vmagent per second. If the limit is exceeded, the ingestion rate will be throttled. * fix changelog * fix review comment (cherry picked from commit `02bccd1eb9`)	2024-03-25 15:42:52 +01:00
Nikolay	3ea0b87399	docs/managed: adds alertmanager configuration examples (#5989 ) * docs/managed: adds alertmanager configuration examples * apply review suggestions (cherry picked from commit `db3709c87d`)	2024-03-25 15:42:52 +01:00
hagen1778	a1b666c35e	docs: add missing API version to VMSingle example Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `93a29fce4e`)	2024-03-25 15:42:51 +01:00
hagen1778	2b05559293	docs: mention Query Analyzer in docs Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `21d9393c9e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-25 15:42:51 +01:00
Yury Molodov	30778e0d0f	vmui: fix the `_time` filter insertion for all queries in VictoriaLogs UI #5920 (#5927 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5920 (cherry picked from commit `46fd0ed693`)	2024-03-25 15:42:33 +01:00
Dmytro Kozlov	9817ecdcc9	app/vmctl: break explore phase in `vm-native` mode by time intervals When `--vm-native-step-interval` is specified, explore phase will be executed within specified intervals. Discovered metric names will be associated with time intervals at which they were discovered. This suppose to reduce number of requests vmctl makes per metric name since it will skip time intervals when metric name didn't exist. This should also reduce probability of exceeding complexity limits for number of selected series in one request during explore phase. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5369 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5f8b91186a`)	2024-03-25 15:42:24 +01:00
hagen1778	c0659800d5	lib/promauth: follow-up `b577413d3b` Convert test result expectations to canonical form. Starting from `b577413d3b` specified header keys are forced into canonical form https://pkg.go.dev/net/http#CanonicalHeaderKey Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e6dd52b04c`)	2024-03-25 15:42:22 +01:00
hagen1778	9fcd34287b	docs: follow-up for `70eaa06f08` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `bc79f7196d`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-25 15:33:11 +01:00
kbweave	1fc3b39b9c	app/vmselect: use GetDeadlineForLabelsAPI for LabelAPI requests (#5992 ) (cherry picked from commit `70eaa06f08`)	2024-03-25 15:31:21 +01:00
Aliaksandr Valialkin	cd222d6502	lib/streamaggr: ignore out of order samples for `last` output This is a follow-up for `6a465f6e29` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931	2024-03-18 01:03:58 +02:00
Aliaksandr Valialkin	5b90106c4b	vendor: run `make vendor-update`	2024-03-18 00:51:52 +02:00
Aliaksandr Valialkin	eecc5e8463	lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search The metricID->metricName entry can remain invisible for search for some time after registering new metricName. This is expected condition. So wait for up to 60 seconds in the hope that the metricID->metricName entry will become visible before deleting all the entries from indexdb, which are associated with the given metricID. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5959 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948 See also `20812008a7`	2024-03-18 00:37:11 +02:00
Aliaksandr Valialkin	05236e7cd4	lib/httputils: rename CAFile -> caFile in order to be consistent with local var naming in Go This is a follow-up for `83e55456e2`	2024-03-17 23:31:53 +02:00
Aliaksandr Valialkin	2f751212a9	Revert "deployment/docs: use lower-case links to VictoriaLogs docs" This reverts commit `a0937b01c1`. Reason for revert: MixedCase links started working again. See, for example, https://docs.victoriametrics.com/VictoriaLogs/querying/#vmui	2024-03-17 23:31:32 +02:00
Aliaksandr Valialkin	111d0aa2bf	app/{vmagent,vminsert}: add an ability to ignore input samples outside the current aggregation interval for stream aggregation See https://docs.victoriametrics.com/stream-aggregation.html#ignoring-old-samples	2024-03-17 23:30:46 +02:00
Aliaksandr Valialkin	e70b644f1f	lib/streamaggr: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs Out of order samples may result in unexpected spikes for these outputs. So it is better to ignore such samples. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931	2024-03-17 23:24:14 +02:00
Aliaksandr Valialkin	1f753a049a	lib/streamaggr: follow-up for `15e33d56f1` - Properly set pushSample.timestamp when flushing de-duplicated samples to stream aggregation This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931 - Re-classify this change as feature instead of bugfix at docs/CHANGELOG.md - Verify de-duplication logic for samples with different timestamps Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5939	2024-03-17 23:23:57 +02:00
Aliaksandr Valialkin	c3dbcc7c0e	docs/CHANGELOG.md: clarify that `-datasource.lookback` commnad-line flag is no-op in the upcoming release Document the solution - to switch to eval_delay option at group config. This is a follow-up for `e80b44f19d`	2024-03-17 23:23:31 +02:00
Aliaksandr Valialkin	b481448c53	docs/CHANGELOG.md: document the bugfix from `cb259116b4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802	2024-03-17 23:23:11 +02:00
Aliaksandr Valialkin	7c4d7dc6dd	lib/promauth: properly set `Host` header in requests to scrape targets. The `Host` header must be set via net/http.Request.Host field, since net/http.Client ignores this header if it is set via Request.Header.Set(). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5969 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5970	2024-03-17 23:22:54 +02:00
Artem Navoiev	8f22c44db6	dashboards: statistic per tenant dashboard use variable for datasource in pie charts Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-03-17 23:22:35 +02:00
hagen1778	efddda945e	app/vmctl: fix arguments order in httputils.TLSConfig follow-up after `9d5bf5ba5d` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 23:22:08 +02:00
Khushi Jain	147a63cafe	app/vmctl: fix the order of arguments in TLS config func (#5972 )	2024-03-17 23:20:19 +02:00
Github Actions	eb82f329cb	Automatic update operator docs from VictoriaMetrics/operator@ae6e1b6 (#5964 )	2024-03-17 23:16:52 +02:00
Daria Karavaieva	dbdbbacf93	docs/vmanomaly: fix 404 links (#5968 )	2024-03-17 23:16:15 +02:00
hagen1778	770dd7affd	docs: mention missing vmalert change for memory usage reduction in 1.97.3 `521f9ffb43` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 23:15:23 +02:00
hagen1778	04dd3aa5b6	deployment/docs: mention other log shippers for VictoriaLogs Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 23:14:35 +02:00
hagen1778	1f8dd125fc	deployment/docs: use lower-case links to VictoriaLogs docs Links with upper-case simply don't work for unknown reason. Once the reason is fixed on docs side, this commit can be reverted. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 23:14:01 +02:00
hagen1778	41a1efbea8	docs: follow-up `15e33d56f1` Update documentation according to changes in deduplication logic. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 23:07:51 +02:00
Andrii Chubatiuk	a58a81d80b	lib/streamaggr: pick sample with bigger timestamp or value on deduplicator (#5939 ) Apply the same deduplication logic as in https://docs.victoriametrics.com/#deduplication This would require more memory for deduplication, since we need to track timestamp for each record. However, deduplication should become more consistent. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643 --------- Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2024-03-17 23:06:37 +02:00
Github Actions	61dc9f5c8d	Automatic update operator docs from VictoriaMetrics/operator@de88675 (#5958 )	2024-03-17 20:48:56 +02:00
Zakhar Bessarab	5f1f1f2982	docs: explicitly mention VMUI is available in cluster (#5955 ) It is confusing for cluster users to find that VMUI is available at vmselect as it is only mentioned in the list of URLs. Explicit mention of vmselect URL in docs will make it easier to discover. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-03-17 20:48:07 +02:00
hagen1778	7b2f19a63e	docs: mention bug investigation in 1.99 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-17 20:46:48 +02:00
Tien M. Nguyen	03fb97d42f	feat: include cluster info in alert CPUThrottlingHigh (#5956 )	2024-03-17 20:46:15 +02:00
Hui Wang	349564fd82	vmalert: deprecate cmd-line flag `-datasource.lookback` (#5877 ) * vmalert: deprecate cmd-line flag `-datasource.lookback` * fix lint * review fixes Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e80b44f19d`)	2024-03-12 16:17:40 +01:00
Aliaksandr Valialkin	abf347ec0f	docs: replace `speed up` with more clear `accelerate` wording	2024-03-12 03:03:15 +02:00
Aliaksandr Valialkin	f92d4609e2	lib/storage: optimize /api/v1/labels and /api/v1/label/.../values when match[] contains metric name Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055	2024-03-12 03:01:47 +02:00
nemobis	14b913eb51	docs: fix typo in `stalenes` (#5950 ) Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2024-03-12 02:59:43 +02:00
Aliaksandr Valialkin	540f65cc49	lib/storage: move the conversion of tag filters to composite tag filters into indexSearch.searchMetricIDsInternal This makes the code less fragile - it is harder to skip the convertToCompositeTagFilterss() call now. While at it, call indexSearch.containsTimeRange() inside indexSearch.searchMetricIDsInternal() in order to quickly terminate search of time series in the old indexdb for new time ranges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055 This is a follow-up for `2d31fd7855`	2024-03-12 02:59:04 +02:00
Github Actions	221f0a000b	Automatic update operator docs from VictoriaMetrics/operator@9b1a6e6 (#5946 )	2024-03-12 02:57:08 +02:00
Aliaksandr Valialkin	293f03f2dd	lib/storage: use composite indexes (metricName, label=value) when searching for matching time series at /api/v1/labels, /api/v1/label/.../values and /api/v1/status/tsdb This should improve query performance when match[], extra_filters[] or extra_label args are passed to these APIs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055	2024-03-12 02:56:35 +02:00
Artem Navoiev	1348a822f8	docs:vmbackup fix typo sped -> speed Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-03-12 02:55:08 +02:00
Zakhar Bessarab	217893921c	docs: fix typo in flags description (#5942 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-03-12 02:45:46 +02:00
Github Actions	b2b034ddb7	Automatic update operator docs from VictoriaMetrics/operator@9caa896 (#5941 )	2024-03-10 13:00:35 +02:00
Github Actions	9d57092a58	Automatic update operator docs from VictoriaMetrics/operator@403a78a (#5932 )	2024-03-10 13:00:22 +02:00
hagen1778	67efc174a8	app/vmauth: properly initialize URLPrefix in tests It is assumed that URLPrefix.busOriginal will be initialized durin Unmarshal of the config. But in tests we set fields manually, so this field never get initialized properly. Fixes the error `panic: runtime error: integer divide by zero` at `vmauth.getLeastLoadedBackendURL`. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `cb1e618a16`)	2024-03-08 21:15:24 +01:00
hagen1778	86abbe5b67	app/vmctl: support TLS configuration for VictoriaMetrics destination VictoriaMetrics destination is specified via `--vm-*` cmd-line flags and is used in opentsdb, influx, prometheus, remote-read modes. updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5426 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `0b7ce70df4`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-08 21:15:23 +01:00
hagen1778	b1fe25792c	app/vmctl: follow-up `b9f7c3169a` * fix typos in flags description * move the change to #tip section in changelog `b9f7c3169a` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `83a8c24281`)	2024-03-08 21:15:09 +01:00
Khushi Jain	98c0909a12	app/vmctl : Provide TLS config options for vm native protocol (#5824 ) Co-authored-by: Khushi Jain <khushi.jain@nokia.com> (cherry picked from commit `b9f7c3169a`)	2024-03-08 21:15:09 +01:00
Aliaksandr Valialkin	b21fb355fa	docs/vmauth.md: update -help output after e08b91baafc95da090f75e9c29a27d8f62a2b76e	2024-03-07 01:37:48 +02:00
Aliaksandr Valialkin	b9788d5cf8	docs/CHANGELOG.md: typo fixes	2024-03-07 01:35:41 +02:00
Aliaksandr Valialkin	d9e3670627	lib/promauth: set the Host header to tlsServerName if itsn't empty If tlsServerName isn't empty, then it is likely the https request is sent to IP instead of hostname. In this case the request will fail, since Go automatically sets the Host header to the IP instead of the desired hostname at tlsServerName. So set the Host header to tlsServerName if itsn't empty. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5802	2024-03-07 01:23:40 +02:00
Aliaksandr Valialkin	e7e9721476	docs/vmauth.md: typo fixes after `7b2b980181`	2024-03-07 01:08:40 +02:00
Aliaksandr Valialkin	93ef217857	app/vmauth: allow discovering backend ips behind shared hostname and spreading load among the discovered ips This is done with the `discover_backend_ips` option at `user` and `url_map` level. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707	2024-03-07 01:02:38 +02:00
Aliaksandr Valialkin	b2e182ca62	app/vmauth: add `src_headers` option at `url_map`, which allows routing incoming requests to different backends depending on request headers	2024-03-06 21:56:42 +02:00
Aliaksandr Valialkin	f5dd9b423e	vendor: run `make vendor-update`	2024-03-06 21:24:55 +02:00
Aliaksandr Valialkin	5bd4325135	app/vmauth: simplify configuration for `src_query_args` Use the shorter form: src_query_args: - arg1=value1 - arg2=value2 instead of src_query_args: - name: arg1 value: value2 - name: arg2 value: value2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878	2024-03-06 21:24:53 +02:00
Aliaksandr Valialkin	1df7e5b69d	deployment/docker: update Go builder from Go1.21.7 to Go1.22.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved	2024-03-06 21:04:44 +02:00
Aliaksandr Valialkin	0929f255f0	docs/vmauth.md: mention that request query args can used for routing decisions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878	2024-03-06 20:58:10 +02:00
Aliaksandr Valialkin	87d59cdfa5	app/vmauth: add ability to route requests based on HTTP query args via src_query_args option See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878	2024-03-06 20:52:42 +02:00
Aliaksandr Valialkin	247d2f3118	app/vmauth: small code cleanup for working with auth tokens	2024-03-06 20:06:25 +02:00
Aliaksandr Valialkin	5817bf1c59	lib/streamaggr: add tests for keep_metric_names and drop_input_labels options	2024-03-06 20:06:23 +02:00
Aliaksandr Valialkin	03c65c9619	app/vmauth: use slices.Contains() instead of hasInt()	2024-03-06 17:36:02 +02:00
Aliaksandr Valialkin	1d7efc4c64	app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at `73f5fb0f0c`	2024-03-06 13:57:53 +02:00
Aliaksandr Valialkin	2711732944	app/vmselect/prometheus: do not drop `match[]` filters if `-search.ignoreExtraFiltersAtLabelsAPI` flag is set The `match[]` filter is mandatory at /api/v1/series, so it mustn't be dropped here. There is no sense in dropping `match[]` filter together with `extra_label` and `extra_filters[]` at /api/v1/labels and /api/v1/label/.../values if -search.ignoreExtraFiltersAtLabelsAPI commnad-line flag is set, since: - the `match[]` filter triggers slow path at these APIs; - the `extra_label` and `extra_filters[]` filters narrow down the number of matched time series, so they improve performance comparing to the case when only `match[]` filter is left, while `extra_label` and `extra_filters[]` filters are dropped. This is a follow-up for `0b7a23a91d`	2024-03-06 13:57:20 +02:00
Daria Karavaieva	98b56476d7	redirect alias (#5934 )	2024-03-06 13:54:48 +02:00
Yury Molodov	56e61bb54b	vmui: improve tracing styles (#5926 ) Improved trace display for better visual separation of branches: * Increased left padding for each element * Added padding for the last element in the branch	2024-03-06 13:54:07 +02:00
hagen1778	a8d42ce4ed	lib/writeconcurrencylimiter: mention dependency on CPU cores for `-maxConcurrentInserts` flag The change also removes misleading `default` value from README for `maxConcurrentInserts` cmd-line flag. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-06 13:44:16 +02:00
Github Actions	cfce708eb9	Automatic update operator docs from VictoriaMetrics/operator@f028fdf (#5929 )	2024-03-06 13:34:37 +02:00
hagen1778	764fc566ff	dashboards: add more context to cluster dashboard panels Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-06 13:34:10 +02:00
hagen1778	a4cdfaea28	docs: clarify deduplication is needed in multi-retention setup Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-06 13:33:40 +02:00
hagen1778	a364de3cfe	lib/writeconcurrencylimiter: mention dependency on CPU cores for `-maxConcurrentInserts` flag The change also removes misleading `default` value from README for `maxConcurrentInserts` cmd-line flag. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `73f5fb0f0c`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-05 18:56:38 +01:00
Aliaksandr Valialkin	27b9e8ed3e	app/{vmagent,vminsert}: add `-streamAggr.dropInputSamples` command-line flag for dropping the specified labels from input samples before deduplication and streaming aggregation	2024-03-05 02:27:27 +02:00
hagen1778	b612c0536e	docs: re-fresh vmctl docs * add recommendation about network bandiwdth between vmctl, source and destination * use more relevant time series selector in examples - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5835 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-05 00:49:48 +02:00
Aliaksandr Valialkin	c38c45d71f	app/{vminsert,vmagent}: allow using -streamAggr.dedupInterval without -streamAggr.config This allows performing online de-duplication of incoming samples	2024-03-05 00:47:23 +02:00
Aliaksandr Valialkin	4352544d61	lib/streamaggr: do not reset aggregation state after the aggregation took longer than the configured interval It is better from user PoV preserving this state until the next flush	2024-03-04 20:03:45 +02:00
Aliaksandr Valialkin	9f81450b38	lib/streamaggr: add missing "s" suffix in the warning message when the de-duplication or aggregation couldnt be finished in a timely manner	2024-03-04 19:38:39 +02:00
Aliaksandr Valialkin	d10932bd99	lib/streamaggr: benchmark only flush routines in BenchmarkDedupAggrFlushSerial and BenchmarkAggregatorsFlushSerial	2024-03-04 19:13:50 +02:00
Aliaksandr Valialkin	36ee08cad4	Revert "lib/streamaggr: do not flush dedup shards in parallel" This reverts commit `eb40395a1c`. Reason for revert: it has been appeared that the performance gain on multiple CPU cores wasn't visible because the benchmark was generating incorrect pushSample.key. See a207e0bf687d65f5198207477248d70c69284296	2024-03-04 19:13:50 +02:00
Aliaksandr Valialkin	9728aaf5d9	lib/streamaggr: properly generate pushSample.key in benchmarks	2024-03-04 19:13:49 +02:00
Aliaksandr Valialkin	93a057e4e6	lib/streamaggr: reduce the number of pointers at "total" aggregation state This should reduce load on GC when scanning heap objects.	2024-03-04 19:13:49 +02:00
Aliaksandr Valialkin	9e00d8ad60	lib/streamaggr: use multiple job label values in BenchmarkAggregatorsPush instead of single value This should make the benchmark closer to production cases	2024-03-04 19:13:48 +02:00
Aliaksandr Valialkin	9773ad200e	lib/streamaggr: use multiple job labels in BenchmarkAggregatorsPush	2024-03-04 19:13:48 +02:00
Github Actions	b530f6bf73	Automatic update operator docs from VictoriaMetrics/operator@c8ff654 (#5918 )	2024-03-04 19:12:59 +02:00
Aliaksandr Valialkin	482560a1f3	lib/streamaggr: do not flush dedup shards in parallel This significantly increases CPU usage on systems with many CPU cores, while doesn't reduce flush latency too much	2024-03-04 17:01:42 +02:00
Aliaksandr Valialkin	d7252fce79	lib/streamaggr: reduce memory allocations when registering new series in deduplication and aggregation structs	2024-03-04 17:01:41 +02:00
Aliaksandr Valialkin	402dc14ec0	lib/streamaggr: make aggregate.runFlusher() more roubst and clear	2024-03-04 17:01:41 +02:00
Aliaksandr Valialkin	2ffef39bb3	lib/streamaggr: properly drop samples on the first incomplete interval Previously samples were dropped on the first incomplete interval and the next complete interval. Also make sure that the de-duplication is performed just before flushing the aggregate state. This should help the case then dedup_interval = interval.	2024-03-04 17:01:40 +02:00
hagen1778	51745ec5ff	dashboards: update links in various panels * use docs.victoriametrics.com instead of github docs * add links to common terms used in VictoriaMetrics Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-04 17:00:54 +02:00
Aliaksandr Valialkin	c2dae136b3	lib/streamaggr: explicitly call resetSeries after flushSeries This makes the code less fragile	2024-03-04 06:23:36 +02:00
Aliaksandr Valialkin	48a425898a	lib/streamaggr: enable time alignment for aggregate flushed to multiples of interval For example, if `interval: 1m`, then data flush occurs at the end of every minute, while `interval: 1h` leads to data flush at the end of every hour. Add `no_align_flush_to_interval` option, which can be used for disabling the alignment.	2024-03-04 06:23:35 +02:00
Aliaksandr Valialkin	3ba9b2225e	docs/stream-aggregation.md: add `troubleshooting` section with solutions for common problems in streaming aggregation	2024-03-04 03:04:59 +02:00
Aliaksandr Valialkin	d80deaeaf4	lib/streamaggr: ignore the first sample in new time series during staleness_interval seconds after the stream aggregation start for total and increase outputs	2024-03-04 03:04:58 +02:00
Aliaksandr Valialkin	5e9cbfd4db	lib/streamaggr: flush dedup state and aggregation state in parallel on all the available CPU cores This should reduce the time needed for aggregation state flush on systems with many CPU cores	2024-03-04 01:22:41 +02:00
Aliaksandr Valialkin	1e741ed6db	lib/streamaggr: add a benchmark for flushing dedup state	2024-03-04 01:22:40 +02:00
Aliaksandr Valialkin	5205972b83	lib/streamaggr: add a benchmark for measuring the performance of aggregator.flush	2024-03-04 01:22:40 +02:00
Aliaksandr Valialkin	8daf7a3f43	lib/streamaggr: add a benchmark for de-duplicating of 1M samples	2024-03-04 01:22:39 +02:00
Aliaksandr Valialkin	d4a425af87	lib/prompbmarshal: use clear() instead of a loop for clearing tss inside ResetTimeSeries()	2024-03-03 23:40:47 +02:00
Aliaksandr Valialkin	b958135677	lib/promutils: optimize LabelsCompressor.Decompress by using a specialized labelsMap struct instead of sync.Map The labelsMap struct employs the fact that label indexes are condensed around 0, so it stores the referred labels in a slice instead of map and uses slice index as label key. This allows increasing the LabelsCompressor.Decompress performance by up to 3x. This also reduces the latency of data flush in stream aggregation.	2024-03-03 23:25:27 +02:00
Aliaksandr Valialkin	492c6c3ff5	docs/CHANGELOG.md: typo fix	2024-03-02 04:52:41 +02:00
Aliaksandr Valialkin	d0e6541f35	docs/stream-aggregation.md: typo fixes	2024-03-02 04:35:37 +02:00
Aliaksandr Valialkin	b912a45220	docs/stream-aggregation.md: remove superflouous output_relabel_configs from the config example for histogram aggregation	2024-03-02 03:36:08 +02:00
Aliaksandr Valialkin	0d5d46f9db	lib/streamaggr: huge pile of changes - Reduce memory usage by up to 5x when de-duplicating samples across big number of time series. - Reduce memory usage by up to 5x when aggregating across big number of output time series. - Add lib/promutils.LabelsCompressor, which is going to be used by other VictoriaMetrics components for reducing memory usage for marshaled []prompbmarshal.Label. - Add `dedup_interval` option at aggregation config, which allows setting individual deduplication intervals per each aggregation. - Add `keep_metric_names` option at aggregation config, which allows keeping the original metric names in the output samples. - Add `unique_samples` output, which counts the number of unique sample values. - Add `increase_prometheus` and `total_prometheus` outputs, which ignore the first sample per each newly encountered time series. - Use 64-bit hashes instead of marshaled labels as map keys when calculating `count_series` output. This makes obsolete https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5579 - Expose various metrics, which may help debugging stream aggregation: - vm_streamaggr_dedup_state_size_bytes - the size of data structures responsible for deduplication - vm_streamaggr_dedup_state_items_count - the number of items in the deduplication data structures - vm_streamaggr_labels_compressor_size_bytes - the size of labels compressor data structures - vm_streamaggr_labels_compressor_items_count - the number of entries in the labels compressor - vm_streamaggr_flush_duration_seconds - a histogram, which shows the duration of stream aggregation flushes - vm_streamaggr_dedup_flush_duration_seconds - a histogram, which shows the duration of deduplication flushes - vm_streamaggr_flush_timeouts_total - counter for timed out stream aggregation flushes, which took longer than the configured interval - vm_streamaggr_dedup_flush_timeouts_total - counter for timed out deduplication flushes, which took longer than the configured dedup_interval - Actualize docs/stream-aggregation.md The memory usage reduction increases CPU usage during stream aggregation by up to 30%. This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5850 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5898	2024-03-02 03:15:43 +02:00
Aliaksandr Valialkin	31f0dc4b97	lib/streamaggr: allow one second aggregation interval	2024-03-01 21:35:43 +02:00
Aliaksandr Valialkin	7533070a52	lib/promrelabel: use clear() function inside CleanLabels()	2024-03-01 21:34:47 +02:00
hagen1778	8ce65e93d0	docs: mention docs link for VictoriaLogs docker env Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-03-01 21:34:15 +02:00
Aliaksandr Valialkin	7559b6eb97	docs: update `-help` output after recent changes in VictoriaMetrics components	2024-03-01 05:31:13 +02:00
Aliaksandr Valialkin	0dbcc07d04	docs/CHANGELOG.md: typo fix	2024-03-01 04:41:39 +02:00
Aliaksandr Valialkin	5f0dd8743c	docs: bump the latest VictoriaMetrics release from v1.98.0 to v1.99.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.99.0	2024-03-01 04:24:11 +02:00
Aliaksandr Valialkin	be1446f720	docs/LTS-releases.md: update latest LTS release to v1.97.3 and v1.93.13 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.3 and https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.13	2024-03-01 04:21:25 +02:00
Aliaksandr Valialkin	2540b8fdf8	deployment: update VictoriaLogs docker image from v0.4.2-victorialogs to v0.5.0-victorialogs See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.5.0-victorialogs	2024-03-01 04:18:59 +02:00
Aliaksandr Valialkin	4543f86827	deployment: update VictoriaMetrics docker image from v1.98.0 to v1.99.0	2024-03-01 04:15:17 +02:00
Aliaksandr Valialkin	d66680fc47	docs/CHANGELOG.md: cut v1.99.0 release	2024-03-01 03:33:44 +02:00
Aliaksandr Valialkin	dc91c07384	docs/CHANGELOG.md: document v1.97.3 LTS release	2024-03-01 03:31:19 +02:00
Aliaksandr Valialkin	e4a1d6774e	docs: consistently use https://docs.victoriametrics.com/lts-releases/ link for LTS releases	2024-03-01 02:52:41 +02:00
Aliaksandr Valialkin	0fac8d6e87	docs/CHANGELOG.md: document v1.93.13 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.13	2024-03-01 02:41:05 +02:00
Aliaksandr Valialkin	bde2de3298	docs/VictoriaLogs/CHANGELOG.md: typo fix: bufix -> bugfix	2024-03-01 02:41:04 +02:00
Aliaksandr Valialkin	052f2177a4	lib/fs: fix GOOS=windows build after `f8baf29b6e`	2024-03-01 01:46:44 +02:00
Aliaksandr Valialkin	9a470239cf	docs/VictoriaLogs/CHANGELOG.md: cut v0.5.0-victorialogs	2024-03-01 01:43:35 +02:00
Aliaksandr Valialkin	334b90598e	app/vlselect/vmui: run `make vmui-logs-update` after `c51031dd70` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5799	2024-03-01 01:33:37 +02:00
Yury Molodov	e00d313333	vmui: add field for log entries limit (#5799 ) * vmui: add field for log entries limit (#5674) * vmui: refactor useFetchLogs * vmui: fix log query encoding --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-03-01 01:33:35 +02:00
Aliaksandr Valialkin	7675bc8f27	vendor: run `make vendor-update`	2024-03-01 00:55:54 +02:00
Aliaksandr Valialkin	d84cdf98da	app/{vmselect,vlselect}/vmui: run `make vmui-update vmui-logs-update` after `e130f29659` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152	2024-03-01 00:51:49 +02:00
Yury Molodov	c71c2b37c5	vmui: add gap display option for charts #5152 (#5862 )	2024-03-01 00:51:48 +02:00
Aliaksandr Valialkin	816202bca7	lib/protoparser/opentelemetry/firehose: verify that the full response is parsed properly in ProcessRequestBody This is a follow-up for `bf9cb84575` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5899	2024-03-01 00:39:47 +02:00
Andrii Chubatiuk	e575fb1aeb	opentelemetry: fix firehose message parsing (#5899 ) Co-authored-by: Andrii Chubatiuk <wachy@Andriis-MBP-2.lan>	2024-03-01 00:24:14 +02:00
Aliaksandr Valialkin	728fb59e0d	docs: update docs after `0b7a23a91d`	2024-03-01 00:18:46 +02:00
Aliaksandr Valialkin	46d92fae3d	app/vmselect/prometheus: ignore match[] additionally to extra_filters[] and extra_label if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set The match[] at /api/v1/labels and /api/v1/label/.../values also may lead to slow requests and high resource usage if it matches big number of time series. So it must be igrnored if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set. This is a follow-up for `fab02faa3f`	2024-03-01 00:18:08 +02:00
Aliaksandr Valialkin	6aea1e5093	app/{vmagent,vminsert}: follow-up for `434a5803e7` Document the /opentelemetry/v1/metrics endpoint instead of /opentelemetry/api/v1/push, since the /v1/metrics suffix is hardcoded in OpenTelemetry protocol specification. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5871	2024-02-29 18:05:12 +02:00
Nikolay	abb773120d	app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path (#5871 ) * app/{vmagent,vminsert}: adds /v1/metrics suffix for opentelemetry route path it must fix compatibility with opentemetry-collector [spec](https://opentelemetry.io/docs/specs/otlp/\#otlphttp-request) this suffix is hard-coded and cannot be changed with collector configuration * Apply suggestions from code review --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-29 17:59:57 +02:00
Github Actions	5e20be845f	Automatic update operator docs from VictoriaMetrics/operator@7e8f811 (#5902 )	2024-02-29 17:53:11 +02:00
Aliaksandr Valialkin	fed04e7f25	app/vmselect/promql: use unsafe.Slice instead of deprecated reflect.SliceHeader	2024-02-29 17:51:16 +02:00
Aliaksandr Valialkin	319d21eddf	app/vmselect/netstorage: usae unsafe.SliceData instead of deprecated reflect.SliceHeader	2024-02-29 17:38:14 +02:00
Aliaksandr Valialkin	01d8bee14c	lib/mergeset: use unsafe.Slice and unsafe.String instead of deprecated reflect.SliceHeader with unsafe conversion from slice header to string header	2024-02-29 17:29:40 +02:00
Aliaksandr Valialkin	99269ea640	lib/bytesutil: use unsafe.String instead of unsafe conversion of slice header to string header	2024-02-29 17:28:04 +02:00
Aliaksandr Valialkin	ddc61e2309	lib/fs: properly handle the case when data=nil is passed to mUnmap	2024-02-29 17:26:26 +02:00
Aliaksandr Valialkin	22acd84019	lib/storage: use unsafe.Slice instead of deprecated reflect.SliceHeader	2024-02-29 17:24:44 +02:00
Aliaksandr Valialkin	a9fb2e91a6	lib/protoparser/csvimport: unse unsafe.Slice instead of deprecated reflect.SliceHeader	2024-02-29 17:20:05 +02:00
Aliaksandr Valialkin	9bc4c51ceb	lib/fs: use unsafe.Slice instead of deprecated reflect.SliceHeader	2024-02-29 17:18:42 +02:00
Aliaksandr Valialkin	4b1a262475	lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader	2024-02-29 17:17:24 +02:00
Aliaksandr Valialkin	3383f73191	lib/bytesutil: make BenchmarkToUnsafeString and BenchmarkToUnsafeBytes more reliable This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5880	2024-02-29 17:12:30 +02:00
helen	74b0605232	Optimize TouUnsafeBytes to make it leaner, more standards-compliant and (#5880 ) slightly faster.	2024-02-29 17:12:04 +02:00
Alexander Marshalov	b8c588a7e2	fixed broken link in docs to google pub sub (#5901 )	2024-02-29 17:11:46 +02:00
Hui Wang	25e454be4c	metricsql: fix label_join() when `dst_label` is equal to one of the `… (#5886 ) * metricsql: fix label_join() when `dst_label` is equal to one of the `src_label` * Update app/vmselect/promql/transform.go * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-29 16:02:48 +02:00
Aliaksandr Valialkin	cfeae6fffb	docs/VictoriaLogs/CHANGELOG.md: document the bugfix at `a5795f533d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5897	2024-02-29 15:34:57 +02:00
XLONG96	88b9088499	lib/logstorage: avoid panic when parsing regex with stream filter (#5897 )	2024-02-29 15:32:25 +02:00
Aliaksandr Valialkin	7310605c27	docs/CHANGELOG.md: clarify the description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5859 This is a follow-up for `3c74aa6b3d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5888	2024-02-29 15:28:52 +02:00
Aliaksandr Valialkin	9d5a385973	app/{vmalert,vmctl}: consistently use http.NewRequestWithContext() instead of http.NewRequest() + req.WithContext()	2024-02-29 15:25:36 +02:00
Aliaksandr Valialkin	0d2d1897c2	app/vmbackupmanager: follow-up after f2885a5c41dfaeecfecba0aceb8ae65b15e15d1c - Document the change at docs/CHANGELOG.md - Give more clear name for maxHealthChecksRetries constant - Improve logging when storage isn't reachable - Use http.NewRequestWithContext() instead of setting up a request without context and then creating a new request with context vi req.WithContext() call. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5859 Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/735	2024-02-29 15:17:57 +02:00
Hui Wang	99acc5bbba	caseStudies: add NetEaseCloudMusic post (#5891 )	2024-02-29 14:58:15 +02:00
Artem Navoiev	3d9f947e42	docs: update statistic per tenant, add use cases section (#5892 ) * docs: update statistic per tenant, add use cases section Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * prettify Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * Update docs/PerTenantStatistic.md --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-29 14:55:55 +02:00
Aliaksandr Valialkin	38b42489c6	docs/anomaly-detection/guides: bump alertmanager version from v0.25.0 to v0.27.0 after `3723c809a1`	2024-02-29 14:51:39 +02:00
Aliaksandr Valialkin	7832d0800e	app/{vminsert,vmagent}: follow-up after `67a55b89a4` - Document the ability to read OpenTelemetry data from Amazon Firehose at docs/CHANGELOG.md - Simplify parsing Firehose data. There is no need in trying to optimize the parsing with fastjson and byte slice tricks, since OpenTelemetry protocol is really slooow because of over-engineering. It is better to write clear code for better maintanability in the future. - Move Firehose parser from /lib/protoparser/firehose to lib/protoparser/opentelemetry/firehose, since it is used only by opentelemetry parser. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5893	2024-02-29 14:47:20 +02:00
Denys Holius	e55d9b7b61	docker-compose: bump alertmanager version from v0.25.0 to the latest v0.27.0 for VictroiaMetrics Single/Cluster/vmanomaly (#5890 ) see https://github.com/prometheus/alertmanager/releases/tag/v0.27.0	2024-02-29 14:47:06 +02:00
Andrii Chubatiuk	60cf0c9656	{vmagent,vminsert}: added firehose http destination opentelemetry data ingestion support (#5893 ) Co-authored-by: Andrii Chubatiuk <wachy@Andriis-MBP-2.lan> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-29 14:46:16 +02:00
Github Actions	33a65d6988	Automatic update operator docs from VictoriaMetrics/operator@9f1c910 (#5895 )	2024-02-29 13:56:31 +02:00
Aliaksandr Valialkin	4e5cbc5dc6	deployment/docker: downgrade Go builder from 1.22.0 to 1.21.7 Go1.22.0 contains the bug https://github.com/golang/go/issues/65705 , which prevents vmagent from normal operation.	2024-02-29 13:52:58 +02:00
Aliaksandr Valialkin	cb9b46be54	app/vmselect/prometheus: add -search.ignoreExtraFiltersAtLabelsAPI command-line flag for ignoring extra_filters and extra_label args at /api/v1/labels, /api/v1/label/.../values and /api/v1/series	2024-02-29 13:00:24 +02:00
Aliaksandr Valialkin	8187244153	lib/streamaggr: make the BenchmarkAggregatorsPushByJobAvg closer to production case with long list of labels per sample	2024-02-29 02:41:48 +02:00
Artem Navoiev	53c1c603d2	dashboards: update statistic per tenant dashbaord. Change to timeseries panel, add churn rate over 24h and query duration, add billing section Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-02-29 02:41:27 +02:00
Hui Wang	d6ecfffa17	chore: add actual request size in error message (#5889 )	2024-02-29 02:40:57 +02:00
lzfhust	2821993c9c	add xiaohongshu in Case studies (#5885 )	2024-02-29 02:40:00 +02:00
Aliaksandr Valialkin	1d6ebe6404	app/vminsert/netstorage: rename storageNode.broken to storageNode.isBroken for the sake of consistency with storageNode.isReadOnly	2024-02-27 14:42:45 +02:00
Aliaksandr Valialkin	be3f5d1c64	app/vminsert/netstorage: log the error message when pending data wasn't sent to vmstorage nodes because they were unavailable at graceful shutdown	2024-02-27 14:15:15 +02:00
Aliaksandr Valialkin	7d619deacc	docs/Release-Guide.md: https://github.com/VictoriaMetrics/victoriametrics-lts-rpm should contain the latest release See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5811	2024-02-27 13:14:25 +02:00
Roman Khavronenko	baa32adc47	deployment: create a separate env for VictoriaLogs (#5857 ) * deployment: create a separate env for VictoriaLogs The new environment consists of the following components: * VictoriaLogs * fluentbit for collecting logs and sending to VictoriaLogs * VictoriaMetrics for scraping and storing metrics from fluentbit and VictoriaLogs * Grafana with VictoriaLogs datasource for monitoring ----------------- The motivation for creating a separate environment is to simplify existing environments and make it easier to update or modify them in future. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-27 13:13:42 +02:00
Aliaksandr Valialkin	05f56c411e	app/vminsert/netstorage: prevent from infinite loop when the re-routing is enabled and all the vmstorage nodes are unavailable	2024-02-25 03:24:51 +02:00
Aliaksandr Valialkin	1109521806	vendor: retain v0.46.0 for github.com/prometheus/common , since v0.48.0 leads to build error The error is: vendor/github.com/prometheus/client_golang/prometheus/testutil/promlint/promlint.go:71:38: undefined: expfmt.FmtText	2024-02-25 02:20:45 +02:00
Aliaksandr Valialkin	b3dbbc22b9	app/vmselect/promql: properly handle args in count_values_over_time() function Prevsiously they were swapped - the first arg should be the label name and the second arg should be label filters This is a follow-up for e389b7b959e8144fdff5075bf7a5a39b2b0c6dd3 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847	2024-02-25 01:48:37 +02:00
Aliaksandr Valialkin	3519c055d5	vendor: run `make vendor-update`	2024-02-24 03:23:56 +02:00
Aliaksandr Valialkin	63d635a5e4	app: consistently use atomic.* types instead of atomic.* functions See `ea9e2b19a5`	2024-02-24 03:06:14 +02:00
Aliaksandr Valialkin	d845edc24b	lib: consistently use atomic.* types instead of atomic.* functions See `ea9e2b19a5`	2024-02-24 02:10:04 +02:00
Aliaksandr Valialkin	61519f6c22	lib/backup/actions: expose vm_backups_downloaded_bytes_total metric in order to be consistent with vm_backups_uploaded_bytes_total metric	2024-02-24 01:14:57 +02:00
Aliaksandr Valialkin	510e3d9cda	lib/backup/actions: update vm_backups_uploaded_bytes_total metric along the file upload instead of after the file upload This solves two issues: 1. The vm_backups_uploaded_bytes_total metric will grow more smoothly 2. This prevents from int overflow at metrics.Counter.Add() when uploading files bigger than 2GiB	2024-02-24 01:08:34 +02:00
Aliaksandr Valialkin	0ac1c533dc	lib/backup/actions: consistently use atomic.* types instead of atomic.* functions See `ea9e2b19a5`	2024-02-24 01:02:37 +02:00
Aliaksandr Valialkin	6fd6d4c2de	lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency See `ea9e2b19a5`	2024-02-24 00:51:03 +02:00
Aliaksandr Valialkin	a1baf25c2e	lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types See `ea9e2b19a5`	2024-02-24 00:33:07 +02:00
Aliaksandr Valialkin	ca1e78bd16	lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types See `ea9e2b19a5`	2024-02-24 00:29:39 +02:00
Aliaksandr Valialkin	d0538d11d3	lib/mergeset: consistently use atomic.* types instead of atomic.* function calls on ordinary types See `ea9e2b19a5`	2024-02-24 00:29:12 +02:00
Aliaksandr Valialkin	92e098012a	lib/logstorage: consistently use atomic.* type for refCount and mustDrop fields in datadb and storage structs in the same way as it is used in lib/storage See `ea9e2b19a5` and `a204fd69f1`	2024-02-24 00:28:56 +02:00
Aliaksandr Valialkin	7fa700a41c	lib/mergeset: consistently use atomic.* type for refCount and mustDrop fields in table struct in the same way as it is used in lib/storage See `ea9e2b19a5` and `a204fd69f1`	2024-02-24 00:28:37 +02:00
Aliaksandr Valialkin	e7dfcdfff6	lib/storage: consistently use atomic.* type for refCount and mustDrop fields in indexDB, table and partition structs See `ea9e2b19a5`	2024-02-24 00:26:26 +02:00
Aliaksandr Valialkin	e2b0cc873b	lib/storage: convert dedupsDuringMerge from uint64 to atomic.Uint64 This should simplify code maintenance by gradually converting to atomic.* types instead of calling atomic.* functions on int and bool types. See `ea9e2b19a5`	2024-02-24 00:25:44 +02:00
Aliaksandr Valialkin	1eb3346ecc	lib/{storage,mergeset}: properly fix 'unaligned 64-bit atomic operation' panic on 32-bit architectures The issue has been introduced in `bace9a2501` The improper fix was in the `d4c0615dcd` , since it fixed the issue just by an accident, because Go comiler aligned the rawRowsShards field by 4-byte boundary inside partition struct. The proper fix is to use atomic.Int64 field - this guarantees that the access to this field won't result in unaligned 64-bit atomic operation. See https://github.com/golang/go/issues/50860 and https://github.com/golang/go/issues/19057	2024-02-24 00:25:08 +02:00
Aliaksandr Valialkin	dc5b1e4dc1	lib/httpserver: return back the default value for -http.connTimeout to 2 minutes It has been appeared that there are VictoriaMetrics users, who rely on the fact that VictoriaMetrics components were closing incoming connections to -httpListenAddr every 2 minutes by default. So let's return back this value by default in order to fix the breaking change made at `d8c1db7953` . See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450 .	2024-02-24 00:20:11 +02:00
Aliaksandr Valialkin	2f0b2b942b	docs/Troubleshooting.md: add `Too much disk space used` chapter`	2024-02-24 00:19:51 +02:00
hagen1778	0ac2df6e66	docs: mention missing change 521f9ffb430edf6aea7720a966b5b079cf15e34e Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-24 00:19:13 +02:00
hagen1778	ab4fae9dc2	lib/storage: cleanup after `d4c0615dcd` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c8d1d2ab72`)	2024-02-23 18:55:40 +01:00
Dmytro Kozlov	eb22083924	lib/storage: fix aligning (#5860 ) (cherry picked from commit `d4c0615dcd`)	2024-02-23 18:55:39 +01:00
Roman Khavronenko	ce4fcd07ce	deployment: add topology visualizations (#5858 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `840ab60111`)	2024-02-23 18:55:39 +01:00
Aliaksandr Valialkin	fe545dfc85	app/vmstorage: remove unused import after `2a5c6e1cd5`	2024-02-23 04:53:42 +02:00
Aliaksandr Valialkin	2a5c6e1cd5	app/vmstorage: deprecate -snapshotCreateTimeout command-line flag Creating snapshot shouldn't time out under normal conditions. The timeout was related to the bug, which has been fixed in `6460475e3b` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551	2024-02-23 04:51:57 +02:00
Aliaksandr Valialkin	42437e05c7	lib/storage: do not drop (date, metricID) entries for the date older than 2 days if samples are ingested at this date Previously the (date, metricID) entries for dates older than the last 2 days were removed. This could lead to slow check for the (date, metricID) entry in the indexdb during ingesting historical data (aka backfilling). The issue has been introduced in `431aa16c8d`	2024-02-23 04:06:54 +02:00
Aliaksandr Valialkin	83217b7473	app/vmselect: add -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries options for fine-tuning CPU and RAM usage for /api/v1/series , /api/v1/labels and /api/v1/label/.../values This commit returns back limits for these endpoints, which have been removed at `5d66ee88bd` , since it has been appeared that missing limits result in high CPU usage, while the introduced concurrency limiter results in failed lightweight requests to these endpoints because of timeout when heavyweight requests are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055	2024-02-23 02:56:58 +02:00
Aliaksandr Valialkin	0e08dee8a1	app/{vmselect,vlselect}/vmui: run `make vmui-update vmui-logs-update` after recent changes to app/vmui	2024-02-23 01:40:57 +02:00
Yury Molodov	140aaafad0	vmui: add a time picker to the "Logs Explorer" page (#5808 ) * vmui: add a time picker to the "Logs Explorer" page #5673 * Update app/vmui/packages/vmui/src/pages/ExploreLogs/hooks/useFetchLogs.ts --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-23 01:38:54 +02:00
Yury Molodov	b2b9f6e900	vmui: fix display Popper.tsx (#5842 ) * vmui: fix display Popper.tsx * vmui/docs: fix display Popper.tsx --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-23 01:33:12 +02:00
Aliaksandr Valialkin	50ea081d75	docs/Single-server-VictoriaMetrics.md: sync with docs/README.md after `79b57f625c`	2024-02-23 01:28:42 +02:00
Aliaksandr Valialkin	fbae1533fc	docs/CHANGELOG.md: document `d68bb658ce` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5833 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5834	2024-02-23 01:26:02 +02:00
Aliaksandr Valialkin	21170e558c	lib/promutils: hide the math.Round() logic inside ParseTimeMsec() function This should prevent from bugs similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801 in the future This is a follow-up for `ce3ec3ff2e`	2024-02-23 01:21:42 +02:00
Aliaksandr Valialkin	dfcbcf4368	lib/mergeset: run `go fmt` after `bace9a2501`	2024-02-23 01:21:31 +02:00
Aliaksandr Valialkin	19032f9913	lib/{mergeset,storage}: convert bufferred items to searchable parts more optimally Do not convert shard items to part when a shard becomes full. Instead, collect multiple full shards and then convert them to a searchable part at once. This reduces the number of searchable parts, which, in turn, should increase query performance, since queries need to scan smaller number of parts.	2024-02-23 01:21:03 +02:00
hagen1778	225fd781a5	docs: re-classify change of default http timeout to bugfix The change introduced in `d8c1db7953 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)` was backported to v1.97.2. Therefore, it is a `bugfix` and should be explicitly mentioned in the changelog of v1.97.2 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-23 01:17:40 +02:00
Nikolay	22762d7a69	app/vmselect: change export/csv timestamp format for rfc3339 to respect milliseconds (#5853 ) * app/vmselect: adds milliseconds to the csv export response for rfc3339 * milliseconds is a standard prescion for VictoriaMetrics query request responses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5837 * app/victoria-metrics: adds tests for csv export/import follow-up after 3541a8d0cf96dd4f8563624c4aab6816615d0756 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2024-02-23 01:16:08 +02:00
Aliaksandr Valialkin	08c5250a7b	lib/storage: handle common case when the number of rows passed to flushRowsToInmemoryParts() doesnt exceed maxRawRowsPerShard	2024-02-23 01:12:18 +02:00
Aliaksandr Valialkin	8669584e9f	lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval Previously the interval between item addition and its conversion to searchable in-memory part could vary significantly because of too coarse per-second precision. Switch from fasttime.UnixTimestamp() to time.Now().UnixMilli() for millisecond precision. It is OK to use time.Now() for tracking the time when buffered items must be converted to searchable in-memory parts, since time.Now() calls aren't located in hot paths. Increase the flush interval for converting buffered samples to searchable in-memory parts from one second to two seconds. This should reduce the number of blocks, which are needed to be processed during high-frequency alerting queries. This, in turn, should reduce CPU usage. While at it, hardcode the maximum size of rawRows shard to 8Mb, since this size gives the optimal data ingestion pefromance according to load tests. This reduces memory usage and CPU usage on systems with big amounts of RAM under high data ingestion rate.	2024-02-23 01:11:57 +02:00
Aliaksandr Valialkin	5f1fa8e7f7	lib/storage: avoid superflouos copy of block header data	2024-02-23 01:11:31 +02:00
Fred Navruzov	c9d8627676	- v1.11 doc updates (#5852 ) - fix dead links	2024-02-23 01:11:05 +02:00
Dan Dascalescu	6b5e8e7089	docs: CSV RFC3339 format uses server timezone (#5839 )	2024-02-23 01:07:55 +02:00
Aliaksandr Valialkin	a982ab6bfb	app/vmstorage: expose vm_snapshots metric, which shows the current number of snapshots While at it, refresh docs about snapshots - https://docs.victoriametrics.com/#how-to-work-with-snapshots	2024-02-23 01:07:04 +02:00
Aliaksandr Valialkin	3f9022bc08	lib/storage: do not pool rawRowsBlock when flushing rawRows to in-memory blocks The pooled rawRowsBlock objects occupies big amounts of memory between flushes, and the flushes are relatively rare. So it is better to don't use the pool and to allocate rawRow blocks on demand. This should reduce the average memory usage between flushes.	2024-02-23 01:06:28 +02:00
Aliaksandr Valialkin	bf07e2ac87	lib/storage: do not keep rawRows buffer across flush() calls The buffer can be quite big under high ingestion rate (e.g. more than 100MB). This leads to increased memory usage between buffer flushes. So it is better to re-create the buffer on every flush in order to reduce memory usage between buffer flushes.	2024-02-23 01:06:09 +02:00
Aliaksandr Valialkin	843f3ec94e	docs/MetricsQL.md: improve text formatting for better readability	2024-02-23 01:05:49 +02:00
Aliaksandr Valialkin	477fdc21aa	app/vmselect/promql: add `count_values_over_time()` MetricsQL function See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847	2024-02-23 01:05:31 +02:00
Aliaksandr Valialkin	65fb54ab8f	app/vmselect/promql: move needSilenceIntervalForRollupFunc from eval.go to rollup.go This should improve maintainability of the code related to rollup functions, since it is located in rollup.go While at it, properly return empty results from holt_winters(), rate_over_sum(), sum2_over_time(), geomean_over_time() and distinct_over_time() when there are no real samples on the selected lookbehind window. Previously the previous sample value was mistakenly returned from these functions.	2024-02-23 01:05:11 +02:00
Alexander Marshalov	8322425364	[lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5814 ) * [lib/promutils, lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801) * fixed tests * fixed test * Revert "fixed test" This reverts commit `8a29764806`. * Revert "fixed tests" This reverts commit `9ce13d1042`. * Revert "[lib/promutils, lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801)" This reverts commit `a7a04bd4` * [lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801) --------- Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-02-23 00:58:26 +02:00
Artem Navoiev	b46014c8ab	docs: change header from h1 to h2 1.97.2. The markdown requires the proper structure in hierachy of title so h1 can not be a child of h1,h2... only be a separate item, in our structure title is the parent h1 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-02-23 00:42:15 +02:00
Github Actions	e2bbab072a	Automatic update operator docs from VictoriaMetrics/operator@d88157b (#5845 )	2024-02-23 00:41:10 +02:00
Github Actions	f08f33cd5a	Automatic update operator docs from VictoriaMetrics/operator@c393852 (#5844 )	2024-02-23 00:40:38 +02:00
Github Actions	786679135b	Automatic update operator docs from VictoriaMetrics/operator@4791fd1 (#5843 )	2024-02-23 00:40:04 +02:00
hagen1778	10fbda60c8	deployment/docker: add comments to components in docker-compose manifests This should help readers to understand interconnectivity between components. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-23 00:39:05 +02:00
Aliaksandr Valialkin	af366a1f97	README.md: sync with docs/Cluster-VictoriaMetrics.md after `e0569a355b`	2024-02-23 00:22:54 +02:00
Anton L	8b7ff0f66e	#5833 Fix Deadlock when using shardByURL of VMAgent (#5834 )	2024-02-22 11:54:53 +02:00
Dan Dascalescu	0c7eda7c88	app/vmselect: simplify wording for `too many samples` error (#5827 ) (cherry picked from commit `17cf031fa1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-20 16:29:11 +01:00
Roman Khavronenko	2e172b9361	vmctl : Provide TLS config options for Open TSDB datasource #5797 (#5832 ) Originally implemented here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5797 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: khushijain21 <khushij393@gmail.com> (cherry picked from commit `bb1279bfc4`)	2024-02-20 16:27:52 +01:00
Daria Karavaieva	550b589790	Vmanomaly Quickstart Fix absolute links (#5831 ) * links fix * typo fix (cherry picked from commit `4034d081f4`)	2024-02-20 16:27:52 +01:00
Daria Karavaieva	a68518c0e0	Vmanomaly QuickStart (#5800 ) * first edit * typo 1 * typo 2 * fixes 3 * fixes 4 * fixes 5 * fixes, cross links * v1.10 config * models why, self-monitoring fix * config next steps * fixes * minor fix (cherry picked from commit `b2baf7d472`)	2024-02-20 16:27:52 +01:00
hagen1778	c065287757	docs: move recent changes to Tip These changes were mistakenly put to existing release Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `dc25c30fdc`)	2024-02-20 16:27:51 +01:00
igorbernstein	b485e40823	deployment/docker: clean up loading of victoriametrics-datasource (#5793 ) Currently the docker-compose examples for loading `victoriametrics-datasource` uses 2 environment variables: - `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` - `GF_DEFAULT_APP_MODE` I believe both of the env vars are trying to achieve the same thing. `GF_DEFAULT_APP_MODE` disables code signing for all plugins and `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` intends to disable code signing for just `victoriametrics-datasource`. Keeping the scope narrowed to just `victoriametrics-datasource` would be preferable in this case. Unfortunately `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` is misspelled. According to [grafana docs](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables), the format is supposed to be `GF_<SectionName>_<KeyName>`. In other words the current env var is missing the section name. This PR proposes to: 1. fix the typo 2. remove the global disablement of code signing Alternatively, if you prefer to keep codesigning disabled globally, please remove `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` env var as it confuses things (cherry picked from commit `cc5a274e4d`)	2024-02-20 16:27:51 +01:00
hagen1778	4474c23aed	app/vmalert: consistently sort groups by name and filename on `/groups` page This should prevent non-deterministic sorting for groups with identical names. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e2dad3a2ac`)	2024-02-20 13:51:31 +01:00
hagen1778	6c63fd831d	app/vmalert: follow-up after `b60dcbe11f` * support case-insensitive search * reflect search condition in URL, so link can be sharable * support filtering on /alerts page * fix collapseAll/expandAll logic to respect only shown entries * add changelog `b60dcbe11f` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `11b03d9fc8`)	2024-02-20 13:35:02 +01:00
Victor Amorim dos Santos	f79abd54b0	vmalert: add filter by group or rule name to UI (#5791 ) Co-authored-by: Yury Molodov <yurymolodov@gmail.com> (cherry picked from commit `b60dcbe11f`)	2024-02-20 13:35:02 +01:00
Yury Molodov	7d15c5abeb	vmui: update package-lock.json (#5822 ) This should address detected security vulnerabilities (cherry picked from commit `524c0a2e07`)	2024-02-20 13:35:02 +01:00
Artem Navoiev	e0569a355b	docs: mention slack inviter and slack channel (#5817 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `5b652bccad`)	2024-02-20 13:35:01 +01:00
Aliaksandr Valialkin	1de59610c8	docs/VictoriaLogs/CHANGELOG.md: document `cafd6f08b3` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5400	2024-02-18 23:18:12 +02:00
Aliaksandr Valialkin	c4b16e3749	docs/VictoriaLogs/CHANGELOG.md: document `333bda8702` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5447	2024-02-18 23:13:30 +02:00
Aliaksandr Valialkin	b58c429044	app/vlselect: follow-up for `451d2abf50` - Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb. See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request. - Document the change at docs/VictoriaLogs/CHANGELOG.md - Document the `limit` query arg at docs/VictoriaLogs/querying/README.md - Make the change less intrusive. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778	2024-02-18 23:06:08 +02:00
Dmytro Kozlov	2d674f98d4	Enable the `limit` query param for the `/select/logsql/query` (#5778 ) * app/vlselect: add limit for logs query * app/vlselect: CHANGELOG.md * app/vlselect: stop search process if limit is reached, update logic, remove default limit * app/vlselect: fix tests * app/vlselect: fix filter tests * app/vlselect: fix tests	2024-02-18 22:59:16 +02:00
Aliaksandr Valialkin	82e38e1627	lib/promscrape: add support for `enable_compression` option in the same way as Prometheus does Updates https://github.com/prometheus/prometheus/pull/13166 Updates https://github.com/prometheus/prometheus/issues/12319 Do not document enable_compression option at docs/sd_configs.md, since vmagent already supports more clear disable_compression option - see https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements	2024-02-18 19:42:09 +02:00
Aliaksandr Valialkin	f0db7d474f	lib/promscrape/discovery/kuma: add support for `client_id` option See https://github.com/prometheus/prometheus/pull/13278	2024-02-18 19:19:55 +02:00
Aliaksandr Valialkin	690f6c833c	vendor: update github.com/VictoriaMetrics/metricsql from v0.72.1 to v0.73.0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5383	2024-02-18 18:43:41 +02:00
Aliaksandr Valialkin	c1a3876e95	docs/MetricsQL.md: properly document how MetricsQL selects the lookbehind window - rate(m) isn't equivalent to rate(m[1i]) when step is smaller than the interval between samples. - default_rollup(m) isn't equivalent to default_rollup(m[1i]) when step is smaller than the interval between samples. These changes have been made in v1.85.3 as a part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483 . See the corresponding commit - `9fa3f1dc57` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5816	2024-02-18 13:48:45 +02:00
Aliaksandr Valialkin	689c0b9817	vendor: update github.com/VictoriaMetrics/metrics from v1.31.0 to v1.32.0	2024-02-18 12:42:29 +02:00
Aliaksandr Valialkin	8053ff9135	docs/LTS-releases.md: cosmetic fixes	2024-02-17 18:09:43 +02:00
Aliaksandr Valialkin	55bba932d4	docs/CHANGELOG.md: document `f8207e33a2`	2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin	cf96c3416d	docs/CHANGELOG.md: add missing `for` in the description of the TLS configuration features for vmctl This is a follow-up for `6a07cb1bdb` and `f973711e56`	2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin	efbead081c	docs/Single-server-VictoriaMetrics.md: enumerate all the VictoriaMetrics components	2024-02-17 17:54:34 +02:00
Aliaksandr Valialkin	9f7c9e9033	docs/LTS-releases.md: add a dedicated page describing LTS lines of releases for VictoriaMetrics	2024-02-17 17:53:56 +02:00
Alexander Marshalov	89e9bfc276	lib/httputils: fixed error message for getting zero duration (#5795 ) (#5812 ) (cherry picked from commit `f8207e33a2`)	2024-02-16 15:31:59 +01:00
hagen1778	e53f53aaf5	app/vmctl: follow-up after `0c293a66ec` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f973711e56`)	2024-02-16 15:31:58 +01:00
Khushi Jain	9ce7f21a63	app/vmctl : support TLS config options for remote read mode (#5798 ) (cherry picked from commit `0c293a66ec`)	2024-02-16 15:31:58 +01:00
hagen1778	025e52adad	app/vmctl: follow-up after `7cd1b7d047` * cleanup code * update docs Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6a07cb1bdb`)	2024-02-16 15:31:58 +01:00
Khushi Jain	02c8b5015c	app/vmctl : support TLS config options for InfluxDB datasource (#5783 ) * vmctl: TLS flags for influx DB * added httputils function * Add changelog and doc --------- Co-authored-by: Khushi Jain <khushi.jain@nokia.com> (cherry picked from commit `7cd1b7d047`)	2024-02-16 15:31:57 +01:00
hagen1778	f4578826b3	dashboards: add legend details to network panels in cluster dash Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ecccd2a1cc`)	2024-02-16 15:31:57 +01:00
Fred Navruzov	d8de87aeb0	docs: vmanomaly - updates of v1.10.0 and model type section (#5813 ) * - apply v1.10 changes - chapter on model types (uni/multivariate and rolling) * - update self-monitoring labels description - fix typos * fix duplicated text and link rendering (cherry picked from commit `172e196ac9`)	2024-02-16 15:31:56 +01:00
hagen1778	5e4732cc2d	docs: update formatting for usage examples - Use `sh` format for examples - Reduce length of progress bars Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3170ad3f44`)	2024-02-16 15:31:56 +01:00
Aliaksandr Valialkin	33b2553c78	app/vmstorage: expose vm_last_partition_parts metrics, which may help identifying performance issues related to the increased number of parts in the last partition	2024-02-15 14:52:53 +02:00
Aliaksandr Valialkin	d4875cccdf	lib/uint64set: go fmt after `c0a9b87f46`	2024-02-15 14:52:53 +02:00
Aliaksandr Valialkin	4e9b70e8b4	lib/mergeset: optimize Set.AddMulti() a bit for len(items) < 10000 This should improve the search speed for time series matching the given label filters	2024-02-15 14:31:00 +02:00
Aliaksandr Valialkin	c89f4c97f3	lib/uint64set: benchmark AddMulti on small number of items, since this case is the most frequent in lib/storage	2024-02-15 14:31:00 +02:00
Aliaksandr Valialkin	6fd9331b7f	docs/CHANGELOG.md: document v1.93.12 LTS release	2024-02-14 20:17:52 +02:00
Aliaksandr Valialkin	c617379789	docs/CHANGELOG.md: document v1.97.2 LTS release	2024-02-14 18:50:59 +02:00
Aliaksandr Valialkin	77a514f2cc	all: update Docker image tag for VictoriaMetrics components from v1.97.1 to v1.98.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.98.0	2024-02-14 17:14:55 +02:00
Aliaksandr Valialkin	06da06dac0	lib/promrelabel: store the original labels before returning them them to promutils.PutLabels() This should reduce memory allocations. This is a follow-up for `b09bd6c42a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389	2024-02-14 16:09:38 +02:00
Aliaksandr Valialkin	acc37ff28a	docs/CHANGELOG.md: cut v1.98.0 release	2024-02-14 16:00:59 +02:00
Aliaksandr Valialkin	e50ebe467a	vendor: run `make vendor-update`	2024-02-14 15:45:05 +02:00
Aliaksandr Valialkin	990a46c478	lib/promrelabel: factor out applyInternal code into ApplyDebug and Apply functions This improves readability and maintanability Also remove memory allocation from SortLabels()	2024-02-14 14:27:44 +02:00
Aliaksandr Valialkin	e16fc81c74	app/vmselect: add missing handler at /select/.../prometheus/vmui/timezone This is a follow-up for `3a26e4d6ec`	2024-02-14 11:18:07 +02:00
Aliaksandr Valialkin	61608b6303	lib/promscrape: avoid copying labels when -promscrape.dropOriginalLabels command-line flag is set This should save some CPU This regression has been introduced in `487f6380d0` when working on https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389	2024-02-14 03:26:32 +02:00
Aliaksandr Valialkin	9df42f000d	docs/CHANGELOG.md: various typo cleanups	2024-02-14 02:45:25 +02:00
Aliaksandr Valialkin	a74dad09ad	app/vmselect/vmui: run `make vmui-update` after `1c9f13d6c7`	2024-02-14 02:36:06 +02:00
Yury Molodov	b08a23c4a5	vmui: improve the context for autocomplete #5736 #5737 #5739 (#5804 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-14 02:36:06 +02:00
Aliaksandr Valialkin	5f2905d120	app/vmselect: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions to MetricsQL See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641	2024-02-13 23:40:30 +02:00
Nikolay	0a2cc0e873	app/vmauth: properly release memory during config reload (#5805 ) * app/vmauth: properly release memory during config reload previously metrics package hold a refrence for channels for users concurrent requests. it case of churn at `name` field of users configuration, new metric was created. But previous one wasn't deleted. It prevented full parsed configuration from being garbace collected. now all config related metrics are bound to corresponding metrics.Set and unregistered during config reload process. It also must fix an issue with incorrect values for current concurrent user requests https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4690 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-13 20:49:57 +02:00
Aliaksandr Valialkin	89aceb2b2f	docs: add link to https://docs.victoriametrics.com/scrape_config_examples/ to docs about configuring target scraping at vmagent and single-node VictoriaMetrics	2024-02-13 20:48:04 +02:00
Aliaksandr Valialkin	2c44ea89fd	docs/vmbackupmanager.md: mention -license command-line flag instead of deprecated -eula	2024-02-13 20:41:33 +02:00
Aliaksandr Valialkin	3f74cadba4	docs/vmauth.md: add `Config reload` chapter, which explains how to reload -auth.config at vmauth Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194	2024-02-13 20:36:59 +02:00
Aliaksandr Valialkin	a92f7a870d	.github/workflows: update actions/cache from v3 to v4 See https://github.com/actions/cache?tab=readme-ov-file#v4	2024-02-13 19:35:46 +02:00
Aliaksandr Valialkin	f6cb2021c3	docs/enterprise.md: remove the mention of deprecated -eula command-line flag	2024-02-13 18:47:07 +02:00
Aliaksandr Valialkin	dcacf209d9	vendor: update github.com/VictoriaMetrics/metricsql from v0.70.1 to v0.71.0 This adds an ability to propagate label filters across label_set() and alias() functions. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827#issuecomment-1654095358	2024-02-13 06:37:09 +02:00
Aliaksandr Valialkin	67091537ae	app/vmagent/remotewrite: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning tls handshake timeout to -remoteWrite.url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699	2024-02-13 02:46:24 +02:00
Aliaksandr Valialkin	84e439c469	docs/vmauth.md: add missing dot	2024-02-13 01:09:11 +02:00
Aliaksandr Valialkin	6bc70a883d	app/vmauth: add support for mTLS-based routing of incoming requests to different backends depending on the subject field in the TLS certificate provided by the user Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1547	2024-02-13 01:04:19 +02:00
Aliaksandr Valialkin	ebcbca0e00	docs/keyConcepts.md: do not duplicate the list of supported data ingestion protocols - just refer to the original list at https://docs.victoriametrics.com/#how-to-import-time-series-data This should simplify keeping docs in sync	2024-02-12 22:55:27 +02:00
Aliaksandr Valialkin	0b2087b6e5	Makefile: run `go mod tidy` with -compat=1.22 after `95222b2079`	2024-02-12 22:36:32 +02:00
Aliaksandr Valialkin	e75f64e993	vendor: run `make vendor-update`	2024-02-12 22:31:41 +02:00
Aliaksandr Valialkin	8793817fbe	docs/vmbackup.md: remove the unneeded -storageDataPath command-line from the example for making server-side copy of the backup	2024-02-12 22:24:41 +02:00
Aliaksandr Valialkin	f5680a6857	all: upgrade Go builder from Go1.21.7 to Go1.22.0 See https://go.dev/doc/go1.22	2024-02-12 22:14:00 +02:00
Aliaksandr Valialkin	99aaa5067f	lib/mergeset: do not panic on too long items passed to Table.AddItems() Instead, log a sample of these long items once per 5 seconds into error log, so users could notice and fix the issue with too long labels or too many labels. Previously this panic could occur in production when ingesting samples with too long labels.	2024-02-12 20:18:19 +02:00
Aliaksandr Valialkin	397bb8771b	lib/mergeset: properly record the firstItem in metaindexRow at blockStreamWriter.WriteBlock The `3c246cdf00` added an optimization where the previous metaindexRow could be saved to disk when the current block header couldn't be added indexBlock because the resulting indexBlock size became too big. This could result in an empty metaindexRow.firstItem for the next metaindexRow.	2024-02-12 20:16:50 +02:00
Aliaksandr Valialkin	838b2275d7	lib/storage: do not append headerData to bsw.indexData if its size exceeds maxBlockSize This is a follow-up optimization after `3c246cdf00`	2024-02-12 20:16:32 +02:00
Artem Navoiev	5815ea4160	docs vmbackupmanager update flags Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-02-12 20:16:00 +02:00
Aliaksandr Valialkin	12cf2d8b44	docs/enterprise.md: remove duplicate Enterprise word in the same sentence	2024-02-12 20:13:30 +02:00
Aliaksandr Valialkin	0f46a49912	docs/Single-server-VictoriaMetrics.md: substitute duplicate cases studies list with the link to the original list	2024-02-12 20:13:01 +02:00
Aliaksandr Valialkin	8ea4b8a3b4	docs: remove misleading `either` from the description of `or` label filters	2024-02-12 20:12:27 +02:00
Github Actions	c952c678b2	Automatic update operator docs from VictoriaMetrics/operator@e261c37 (#5788 )	2024-02-12 20:11:07 +02:00
Roman Khavronenko	433c3726b2	app/vmalert: support filtering for /api/v1/rule like Prometheus does (#5787 ) Follow-up after `62e5e2a4c8` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8850c7431d`)	2024-02-09 14:36:15 +01:00
Victor Amorim dos Santos	56b1d8e9ed	app/vmalert: support `type` param for filtering /api/v1/rules response by rule type (#5749 ) Co-authored-by: Hui Wang <haley@victoriametrics.com> (cherry picked from commit `62e5e2a4c8`)	2024-02-09 14:36:14 +01:00
Aliaksandr Valialkin	46de5d7eb1	docs: update docs after `ae8a867924` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470	2024-02-09 04:20:09 +02:00
Aliaksandr Valialkin	ae12ac69ba	lib/snapshot: move Time, Validate and NewName into lib/snapshot/snapshotutil package This allows removing importing unneeded command-line flags into binaries, which import lib/storage, which, in turn, was importing lib/snapshot in order to use Time, Validate and NewName functions. This is a follow-up for `83e55456e2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5738	2024-02-09 04:19:30 +02:00
Aliaksandr Valialkin	cf64597878	all: add support for specifying multiple -httpListenAddr options	2024-02-09 03:22:49 +02:00
Aliaksandr Valialkin	5bbf95f9b4	docs/CHANGELOG.md: typo fixes	2024-02-08 21:18:52 +02:00
Aliaksandr Valialkin	ae7da12280	lib/httpserver: do not close client connections every 2 minutes by default Closing client connections every 2 minutes doesn't help load balancing - this just leads to "jumpy" connections between multiple backend servers, e.g. the load isn't spread evenly among backend servers, and instead jumps between the servers every 2 minutes. It is still possible periodically closing client connections by specifying non-zero -http.connTimeout command-line flag. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1636997037 This is a follow-up for `d387da142e`	2024-02-08 21:10:54 +02:00
Aliaksandr Valialkin	9216d00a6c	docs/{vmagent,vmalert}: mention that /-/reload endpoint may be protected with -reloadAuthKey command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1172	2024-02-08 18:49:21 +02:00
Aliaksandr Valialkin	0d313dda8a	docs/Cluster-VictoriaMetrics.md: document that /api/v1/query?series_lector[d] returns raw samples Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148	2024-02-08 18:32:47 +02:00
Aliaksandr Valialkin	fc8328a87f	docs: sync with enterprise branch	2024-02-08 17:25:26 +02:00
Aliaksandr Valialkin	428109a427	docs: update -help output after `83e55456e2`	2024-02-08 17:22:42 +02:00
Artem Navoiev	fbe3f51fee	docs: change ndjson links to https://jsonlines.org/ as original one was hacked (#5782 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-02-08 17:22:31 +02:00
hagen1778	2aa86313bc	docs: sync flags description for vmbackup and vmbackupmanager Sync description after changes in `83e55456e2` Remove extra text in flags description for vmbackupmanager as it breaks layout when rendered at docs.victoriametrics.com Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-08 17:12:53 +02:00
Aliaksandr Valialkin	44052f4fd5	docs/CHANGELOG.md: add a link to docs describing -disableReroutingOnUnavailable command-line flag	2024-02-08 17:05:27 +02:00
hagen1778	a2063e0e7a	docs: follow-up after `83e55456e2` `83e55456e2` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e1926f286b`)	2024-02-08 15:58:34 +01:00
Khushi Jain	a076cb4a93	app/vmbackup: support client-side TLS configuration for create/delete snapshot API (#5738 ) (cherry picked from commit `83e55456e2`)	2024-02-08 15:58:34 +01:00
Aliaksandr Valialkin	d0b55e359a	docs/vmagent.md: add `debugging scrape targets` chapter	2024-02-08 16:32:54 +02:00
hagen1778	70077b0da1	deployment/docker: update README with ToC The change also moves commands for starting/stopping env to corresponding sections. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-08 16:32:53 +02:00
Aliaksandr Valialkin	582d431df2	app/victoria-metrics: properly send staleness markers on victoriametrics shutdown if -selfScrapeInterval > 0 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/943 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2024-02-08 15:30:40 +02:00
Aliaksandr Valialkin	202d8e2c40	docs: update -help output after `61d9df4c36` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834	2024-02-08 14:50:56 +02:00
Aliaksandr Valialkin	b18e608016	app/vmselect: add ability to reset rollup result cache on startup by passing -search.resetRollupResultCacheOnStartup command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834	2024-02-08 14:42:15 +02:00
Aliaksandr Valialkin	1856c9fcc1	lib/mergeset: add a test for too long item passed to Table.AddItems()	2024-02-08 14:14:23 +02:00
Aliaksandr Valialkin	d2a846eddd	lib/mergeset: typo fix: indexdb/indexBlock -> indexdb/indexBlocks	2024-02-08 14:14:23 +02:00
Aliaksandr Valialkin	950b126a09	lib/{storage,mergeset}: do not create index blocks with sizes exceeding 64Kb in common case This should reduce memory fragmentation and memory usage for indexdb/indexBlocks and storage/indexBlocks caches	2024-02-08 14:14:22 +02:00
Aliaksandr Valialkin	1c3eac5c1e	lib/mergeset: verify that the index block for in-memory part doesnt exceed the 3*maxIndexBlockSize	2024-02-08 14:14:22 +02:00
Aliaksandr Valialkin	9a3a88b321	lib/mergeset: do not store commonPrefix in blockHeader if the block contains only a single item There is no sense in storing commonPrefix for blockHeader containing only a single item, since this only increases blockHeader size without any benefits.	2024-02-08 14:14:22 +02:00
Aliaksandr Valialkin	84a3f4719d	docs/CHANGELOG.md: clarify the bugfix description for `e1bf8440eb`	2024-02-08 13:01:26 +02:00
Aliaksandr Valialkin	ae2a9c8195	lib/mergeset: prevent from possible `too big indexBlockSize` panic This panic could occur when samples with too long label values are ingested into VictoriaMetrics. This could result in too long fistItem and commonPrefix values at blockHeader (up to 64kb each). This may inflate the maximum index block size by 4 * maxIndexBlockSize.	2024-02-08 12:55:58 +02:00
hagen1778	9b173c2f01	dashboards: follow-up `4369bc1df2` * add more details to changelog * simplify panels description * remove capacity planning recommendation, as it proves it incompetent Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-08 12:55:42 +02:00
Hui Wang	0cd0ddc1c1	deployment/dashboards: fix `Storage full ETA` panels (#5747 ) During background downsampling, rate(vm_deduplicated_samples_total{type="merge"}) could be much bigger than rate(vm_rows_added_to_storage_total) and it could last quite some time, which causes negative values of Storage full ETA and confuses users, see playground. Instead of trying to get more accurate results during downsampling, I think it's ok to ignore vm_deduplicated_samples_total at all, it's more reasonable to see Storage full ETA increase after downsampling. --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-08 12:54:31 +02:00
Aliaksandr Valialkin	def5573f92	app/vmselect/promql: properly handle precision errors in rollup functions changes(), increases_over_time() and resets() shouldn't take into account value changes, which may occur because of precision errors. The maximum guaranteed precision for raw samples stored in VictoriaMetrics is 12 decimal digits. So do not count relative changes for values if they are smaller than 1e-12 comparing to the value. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767	2024-02-08 02:33:17 +02:00
Aliaksandr Valialkin	f4252cd9aa	docs/CHANGELOG.md: properly document the change at `b74006e2ca` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774	2024-02-07 22:06:37 +02:00
Nihal	95287d269b	[vmsingle/vminsert]: change http success response code to 200 for -/reload request handler (#5776 ) * change vmsingle's response code to 200 for reload request handler Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * change vmsingle's response code to 200 for the reload request handler Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * change vmsingle's response code to 200 for the reload request handler. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5774 Signed-off-by: Syed Nihal <syed.nihal@nokia.com> --------- Signed-off-by: Syed Nihal <syed.nihal@nokia.com>	2024-02-07 22:06:24 +02:00
Github Actions	3c069607b3	Automatic update operator docs from VictoriaMetrics/operator@38dfad5 (#5772 )	2024-02-07 21:55:11 +02:00
Aliaksandr Valialkin	ec02e9ba19	lib/protoparser/datadogsketches: use math.RoundToEven() for calculating the rank The original code uses this function - see `48d52eeea6/pkg/quantile/sparse.go (L138)` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5775	2024-02-07 21:45:05 +02:00
Aliaksandr Valialkin	28fffdfcc7	lib/protoparser/datadogsketches: add more permalinks to the original source code These permalinks should help verifying the correctness of the code This is a follow-up after `07213f4e0c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5775	2024-02-07 21:45:05 +02:00
Andrii Chubatiuk	3aa439a618	added ddsketch permalink (#5775 ) Co-authored-by: Andrew Chubatiuk <andrew.chubatiuk@motional.com>	2024-02-07 21:45:04 +02:00
Aliaksandr Valialkin	5d9e0ab71e	docs/CHANGELOG.md: support empty command-line flag values in short array notation For example, -fooDuration=',10s,' is now supported - it sets three command-line flag values: - the first and the last one are set to the default value for `-fooDuration` - the second one is set to 10s	2024-02-07 20:55:01 +02:00
Aliaksandr Valialkin	949481fe48	docs/relabeling.md: add examples on how to rename scraped metrics and how to add/change labels in scraped metrics	2024-02-07 20:29:04 +02:00
Aliaksandr Valialkin	caf706fcc0	all: update Go builder from Go1.21.6 to Go1.21.7 See https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved	2024-02-07 04:01:05 +02:00
Aliaksandr Valialkin	18e1dfd7a8	docs/scrape_config_examples.md: add an example for scraping cadvisor in Kubernetes	2024-02-07 03:23:01 +02:00
Aliaksandr Valialkin	cc6f05b117	app/vminsert: fix the code after `c634859c4f`	2024-02-07 02:08:34 +02:00
Aliaksandr Valialkin	750ddeef54	app/{vmselect,vlselect}/vmui: run `make vmui-update vmui-logs-update` after the recent changes to app/vmui This is a follow-up for the following commits: - `dcbdbc760e` - `a81ccbd749` - `65b8002aeb`	2024-02-07 01:49:45 +02:00
Aliaksandr Valialkin	898414856f	docs/Cluster-VictoriaMetrics.md: document the /datadog/api/beta/sketches endpoint This is a follow-up for `a1d1ccd6f2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584	2024-02-07 01:36:08 +02:00
Aliaksandr Valialkin	82f4e4e070	app/{vmagent,vminsert}: follow-up after `a1d1ccd6f2` - Document the change at docs/CHANGELOG.md - Copy changes from docs/Single-server-VictoriaMetrics.md to README.md - Add missing handler for processing multitenant requests ( https://docs.victoriametrics.com/vmagent/#multitenancy ) - Substitute github.com/stretchr/testify dependency with 3 lines of code in the added tests - Comment unclear code at lib/protoparser/datadogsketches/parser.go , so @AndrewChubatiuk could update it and add permalinks to the original source code there. - Various code cleanups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3091	2024-02-07 01:31:52 +02:00
Daria Karavaieva	0ff3e6f3bc	Vmanomaly Guide - dashboard and query change (#5771 ) * dashboard fix * query fix * changed screenshots * minor fixes	2024-02-07 01:31:15 +02:00
hagen1778	eb90f2170a	docs/vmalert: mention `step` param when comapring rule results to raw queries Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-07 01:30:32 +02:00
Andrii Chubatiuk	c634859c4f	support datadog /api/beta/sketches API (#5584 ) Co-authored-by: Andrew Chubatiuk <andrew.chubatiuk@motional.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-07 01:30:00 +02:00
Yury Molodov	5778acf9eb	vmui: improve select component functionality (#5755 ) * vmui: fix select closing on click outside (#5728) * vmui: clear entered text in select after selecting a value (#5727) --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-06 22:50:28 +02:00
Yury Molodov	0b5f5d456c	vmui: fix handling invalid timezone (#5758 ) * vmui: fix handling invalid timezone (#5732) * vmui: switch browser timezone flag to isValid --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-06 22:48:00 +02:00
Aliaksandr Valialkin	293617028d	lib/storage: move fixupTimestamps() call to Block.Init() This is a follow-up for `0bf7921721`	2024-02-06 22:44:09 +02:00
Zakhar Bessarab	fdbc44d813	lib/storage/raw_row: properly initialize TS for tmp blocks (#5762 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-02-06 22:44:08 +02:00
Yury Molodov	0cf17068b8	vmui: fix graph dragging (#5769 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-06 22:32:43 +02:00
Aliaksandr Valialkin	ebd0f1b72e	vendor: update github.com/VictoriaMetrics/metricsql from v0.70.0 to v0.70.1 This should help with the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5604	2024-02-06 21:52:55 +02:00
Aliaksandr Valialkin	e19b53748a	lib/fs: lazily open the file at ReaderAt on the first access This should significantly reduce the number of open ReaderAt files on VictoriaMetrics and VictoriaLogs startup. The open files can be tracked via vm_fs_readers metric	2024-02-06 21:10:00 +02:00
Aliaksandr Valialkin	dc340a83a9	docs/scrape_config_examples.md: document that full urls can be used as in the `targets` list	2024-02-06 19:49:23 +02:00
Aliaksandr Valialkin	2fec67632f	docs/scrape_config_examples.md: add missing `to specify` phrase	2024-02-06 19:43:30 +02:00
Aliaksandr Valialkin	0504771db1	docs/CHANGELOG.md: add link to mTLS feature request	2024-02-06 19:32:51 +02:00
Aliaksandr Valialkin	bace92fab6	lib/httpserver: add support for mTLS for requests to -httpListenAddr	2024-02-06 17:47:27 +02:00
Aliaksandr Valialkin	79990cc89b	docs/CHANGELOG.md: move descriptions for recently added features from `v1.97.1` to `tip`	2024-02-06 16:31:44 +02:00
Aliaksandr Valialkin	22524353ed	docs/scrape_config_examples.md: add examples for typical scrape_config usage	2024-02-06 16:01:34 +02:00
Aliaksandr Valialkin	01312003be	docs/Cluster-VictoriaMetrics.md: add a warning that -disableReroutingOnUnavailable command-line flag may result in long pause for data ingestion when some of vmstorage nodes are unavailable for long time	2024-02-06 16:01:07 +02:00
Aliaksandr Valialkin	bffcb40537	docs/enterprise.md: prioritize enterprise technical support over other enterprise features	2024-02-06 16:00:21 +02:00
hagen1778	8bc18b5a4c	docs/vmalert: update `limit` description Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-06 16:00:02 +02:00
hagen1778	702cc4514a	docs/vmalert: mention `limit` option in group params This param was supported for long time but was missing in the docs. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-06 15:59:28 +02:00
Fred Navruzov	eaf75a565f	update changelog with 1.0.0 helm chart ref (#5763 )	2024-02-06 15:58:49 +02:00
Aliaksandr Valialkin	2bad79a97e	docs/sd_configs.md: improve readability a bit	2024-02-05 15:33:23 +02:00
Github Actions	c0b2ccb6ca	Automatic update operator docs from VictoriaMetrics/operator@bfc521d (#5761 ) Co-authored-by: Hui Wang <haley@victoriametrics.com>	2024-02-05 15:33:22 +02:00
Aliaksandr Valialkin	63a43331a3	docs/Cluster-VictoriaMetrics.md: document -disableReroutingOnUnavailable command-line flag This is a follow-up for `88f0d1572e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713	2024-02-05 15:17:09 +02:00
Muxa1L	88f0d1572e	Add flag to vminsert to disable rerouting when some of storage nodes are unavailable (#5713 ) * Flag to disable rerouting from unavailable storage nodes * Update netstorage.go * Fix fmt for netstorage.go	2024-02-05 12:46:57 +00:00
Aliaksandr Valialkin	67d166f181	docs/Cluster-VictoriaMetrics.md: move the `Improve re-routing performance during restart` to more appropriate place Previoulsy it was mistakenly inserted between `No downtime strategy` and 'Minimum downtime strategy' chapters. This is a follow-up for `37997abd14` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5293	2024-02-05 14:44:06 +02:00
Aliaksandr Valialkin	f222cf9200	lib/cgroup: remove SetGOGC() function GOGC can be already set via environment variable. There is no need in adding new approaches for setting the GOGC (such as command-line flag), since they complicate operations.	2024-02-05 12:13:08 +02:00
Aliaksandr Valialkin	e2c58a9db1	docs/Cluster-VictoriaMetrics.md: mention -metrics.exposeMetadata command-line flag in `Monitoring` section	2024-02-05 11:48:05 +02:00
Aliaksandr Valialkin	103f527037	docs/CHANGELOG.md: add a link to all VictoriaMetrics dashboards for Grafana	2024-02-05 11:45:39 +02:00
Aliaksandr Valialkin	6af029fd9b	docs/Troubleshooting.md: clarify the section about GOGC tuning This is a follow-up for `487a94565b`	2024-02-05 11:45:39 +02:00
Aliaksandr Valialkin	b105a5a1f5	docs/CHANGELOG.md: add missing links to the corresponding dashboards	2024-02-05 11:45:06 +02:00
Aliaksandr Valialkin	19f9203c7a	docs: mention -metrics.exposeMetadata command-line flag in `Monitoring` sections This is a follow-up for `326a77c697`	2024-02-05 11:44:39 +02:00
Artem Navoiev	e3138a356c	docs: fix aliasis for url-example Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-02-05 11:43:58 +02:00
Fred Navruzov	be41ffb925	docs: vmanomaly - fix 1.9.2 references in pull cmd and docs (#5756 ) * fix 1.9.2 references in pull cmd and docs * better readability of upgrade note	2024-02-05 11:43:41 +02:00
hagen1778	bdbab7bed5	dashboards/all: add new panel `CPU spent on GC` It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-05 11:42:28 +02:00
hagen1778	2206309439	dashboards: add `Targets scraped/s` A new stat panel shows the number of targets scraped by the vmagent per-second. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-05 10:51:35 +02:00
hagen1778	3dab94a6c1	dashboards: update to grafana/grafana:10.3.1 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-05 10:50:36 +02:00
hagen1778	39e9a0848c	deployment/docker: bump grafana version to grafana/grafana:10.3.1 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-02-05 10:49:52 +02:00
Aliaksandr Valialkin	8148cc52c9	lib/prompbmarshal: code cleanup after `8aaa828ba3`	2024-02-01 21:41:10 +02:00
Aliaksandr Valialkin	7a9f0b32a2	app/vmselect/netstorage: prevent from disk write IO when closing temporary files Remove temporary file before closing it in order to signal the OS that it shouldn't store the file contents from page cache to disk when the file is closed. Gracefully handle the case when the file cannot be removed before being closed - in this case remove the file after closing it. This allows working on Windows. Also remove superflouos opening of temporary file for reading - re-use already opened file handle for writing. This is a follow-up for `9b1e002287` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4020 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2024-02-01 19:54:48 +02:00
Aliaksandr Valialkin	9f23231926	docs/CHANGELOG.md: document v1.93.11 LTS release	2024-02-01 18:21:35 +02:00
Aliaksandr Valialkin	d63bc81f29	docs/CHANGELOG.md: document v1.87.14 LTS release	2024-02-01 17:09:03 +02:00
Aliaksandr Valialkin	24437b6d70	all: update VictoriaMetrics Docker image from v1.97.0 to v1.97.1 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.1	2024-02-01 16:15:14 +02:00
Aliaksandr Valialkin	765917ae6a	vendor: run `make vendor-update`	2024-02-01 15:24:53 +02:00
Aliaksandr Valialkin	dab3da9ee8	docs: mark v1.97.x as long-term support release	2024-02-01 15:16:52 +02:00
Aliaksandr Valialkin	e1ceb64432	docs/CHANGELOG.md: cut v1.97.1	2024-02-01 15:08:53 +02:00
Aliaksandr Valialkin	eff4dc3685	docs: sync `-help` output after recent changes	2024-02-01 15:06:19 +02:00
Aliaksandr Valialkin	c5f2a2b91f	app/vmselect: add missing whitespace into the description for -vmui.defaultTimezone command-line flag This is a follow-up for `eb6def0695` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5611	2024-02-01 14:49:48 +02:00
Aliaksandr Valialkin	d8f508cc21	docs/Single-server-VictoriaMetrics.md: clarify `Security` chapter a bit	2024-02-01 14:45:10 +02:00
Aliaksandr Valialkin	ee327cfd5d	docs/Single-server-VictoriaMetrics.md: run `make docs-sync` after `49d5e7fef5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5262	2024-02-01 14:42:07 +02:00
Dima Lazerka	d561f506cd	Improve docs on security http headers (#5262 ) * Improve docs on security http headers * Apply suggestions from code review --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-02-01 14:40:57 +02:00
noodles2hg	60a8e59366	lib/logstorage: proper exit during block search (#5400 )	2024-02-01 14:11:20 +02:00
Jiajing LU	9c75e3ee15	count inmemoryParts that have not been taken for merge (#5447 )	2024-02-01 14:07:13 +02:00
dependabot[bot]	07e6342915	build(deps): bump codecov/codecov-action from 3 to 4 (#5745 ) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/v3...v4) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-02-01 14:02:06 +02:00
Aliaksandr Valialkin	6c56f49f9c	lib/prompbmarshal: return back custom protobuf marshaler for lib/prompbmarshal.WriteRequest The easyproto-based marshaler is 2x slower than the previous custom marshaler, so let's stick with it. This improves the performance for sending data to remote storage at vmagent and reduces CPU usage to pre-v1.97.0 levels.	2024-02-01 06:34:46 +02:00
Aliaksandr Valialkin	faeabfc730	lib/encoding: follow-up for `49e3665d6d` Improve performance for typical cases of varint marshaling / unmarshaling further. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721	2024-02-01 05:38:58 +02:00
Fuchun Zhang	78af9b3e30	make encoding.MarshalVarInt64s faster (#5721 ) * make encoding.MarshalVarInt64s faster * add fast path for MarshalVarInt64s * make UnmarshalVarUint64s faster * remove comment	2024-02-01 03:33:59 +00:00
Aliaksandr Valialkin	eee210810e	lib/encoding: added benchmarks for marshaling / unmarshaling of varints This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721	2024-02-01 05:11:35 +02:00
helen	99ea84f0fd	clean unused code (#5735 ) Signed-off-by: helen <haitao.zhang@daocloud.io>	2024-01-31 19:51:35 +02:00
Aliaksandr Valialkin	cc626ae3b5	lib/promauth: follow-up for `fca3b14b7b` - Simplify the code for handling BasicAuthConfig at lib/promauth/config.go - Move the description of the change into correct place at docs/CHANGELOG.md - Put tests for username in front of tests for password at lib/promauth/config_test.go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5720 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511	2024-01-31 19:47:53 +02:00
Nihal	bcd094ac8b	Support for username_file in scrape config (basic_auth) similar to Prometheus for having config compatibility (#5720 ) * adding support for username_file in basic_auth of scrape config Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * adding support for username_file in basic_auth of scrape config. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5511 Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * adding support for username_file in basic_auth of scrape config Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * adding support for username_file in basic_auth of scrape config Signed-off-by: Syed Nihal <syed.nihal@nokia.com> * adding support for username_file in basic_auth of scrape config Signed-off-by: Syed Nihal <syed.nihal@nokia.com> --------- Signed-off-by: Syed Nihal <syed.nihal@nokia.com>	2024-01-31 19:47:50 +02:00
Aliaksandr Valialkin	4f5cb17042	app/vmselect/netstorage: properly handle the case when an empty brsPool points to the end of brs.brs This case is possible after a new brsPool is allocated. The fix is to verify whether len(brsPool) >= len(brs.brs) before trying to append a new item to brsPool and sharing its contents with brs.brs. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5733	2024-01-31 10:31:51 +02:00
hagen1778	151247c9b9	dashboards/single: fix typo in query for `version` annotation The typo falsely produced many version change events. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-31 10:28:24 +02:00
Aliaksandr Valialkin	2033fe4caf	app/vmselect/promql: really keep metric names when keep_metric_names modifier is applied to binary operator Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556	2024-01-31 02:33:06 +02:00
Aliaksandr Valialkin	a0043e214f	docs/vmauth.md: typo fix after `68be182075`	2024-01-31 00:13:56 +02:00
Aliaksandr Valialkin	f1effc1133	all: update VictoriaMetrics Docker image from v1.96.0 to v1.97.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0	2024-01-30 23:37:42 +02:00
Aliaksandr Valialkin	6d766abde7	deployment/docker: upgrade base Docker image from Alpine 3.19.0 to 3.19.1 See https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html	2024-01-30 22:47:29 +02:00
Aliaksandr Valialkin	966f1ad07f	docs/CHANGELOG.md: cut v1.97.0	2024-01-30 22:45:10 +02:00
Dan Dascalescu	d2c38e2588	docs: t...over_time functions return fractional seconds (#5715 ) * docs: t...over_time functions return fractional seconds * Apply suggestions from code review --------- Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-30 22:20:16 +02:00
Roman Khavronenko	02e609b141	app/vmselect: set proper timestamp for cached instant responses (#5723 ) * app/vmselect: set proper timestamp for cached instant responses The change updates `getSumInstantValues` to prefer timestamp from the most recent results. Before, timestamp from cached series was used. The old behavior had negative impact on recording rules as they were getting responses with shifted timestamps in past. Subsequent recording or alerting rules fetching results of these recording rules could get no result due to staleness interval. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659 Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-30 22:20:16 +02:00
Aliaksandr Valialkin	3164f7526c	app/vmselect/vmui: run `make vmui-update` after `81b5db04f6`	2024-01-30 21:13:14 +02:00
Yury Molodov	587c5fec2d	vmui: add the ability to expand all tracing entries (#5677 ) (#5726 )	2024-01-30 21:13:13 +02:00
Github Actions	317a2551ee	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@40e4e15 (#5729 )	2024-01-30 21:07:50 +02:00
Aliaksandr Valialkin	1a071c5228	docs/CHANGELOG.md: document the enhancement, which reduces initial memory usage when `vmagent` scrapes targets with large responses Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567	2024-01-30 20:51:29 +02:00
Aliaksandr Valialkin	dc9f4f0907	docs/CHANGELOG.md: refer to the related pull request for the bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2024-01-30 20:21:51 +02:00
Aliaksandr Valialkin	16787415b3	docs/CHANGELOG.md: document the bugfix addressed by the commit `bc7cf4950b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2024-01-30 20:16:32 +02:00
Aliaksandr Valialkin	ccb1143823	vendor: run `make vendor-update`	2024-01-30 18:47:30 +02:00
Aliaksandr Valialkin	c5deb226d4	app/vmselect/vmui: run `make vmui-update` after 6e8995cfb92fb5a87fc6ad78609bf9ea5e0e712f	2024-01-30 18:45:52 +02:00
Artem Navoiev	17a4d310d0	docs: add alias for keyconcepts Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-30 18:39:59 +02:00
Aliaksandr Valialkin	09c388a8e4	lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode While fasthttp.Client uses less CPU and RAM when scraping targets with small responses (up to 10K metrics), it doesn't work well when scraping targets with big responses such as kube-state-metrics. In this case it could use big amounts of additional memory comparing to net/http.Client, since fasthttp.Client reads the full response in memory and then tries re-using the large buffer for further scrapes. Additionally, fasthttp.Client-based scraping had various issues with proxying, redirects and scrape timeouts like the following ones: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5425 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017 This should help reducing memory usage for the case when target returns big response and this response is scraped by fasthttp.Client at first before switching to stream parsing mode for subsequent scrapes. Now the switch to stream parsing mode is performed on the first scrape after reading the response body in memory and noticing that its size exceeds the value passed to -promscrape.minResponseSizeForStreamParse command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567 Overrides https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931	2024-01-30 18:39:55 +02:00
Aliaksandr Valialkin	645365b2d1	lib/promscrape: fix BenchmarkScrapeWorkScrapeInternal, which has been broken by the commit `65bc460323`	2024-01-30 16:07:40 +02:00
Yury Molodov	8958fb78ad	vmui: fix `Enter` key in query field (#5667 ) (#5717 ) (cherry picked from commit `7007c6a760`)	2024-01-30 14:45:47 +01:00
Aliaksandr Valialkin	3b18659487	app/vmagent/remotewrite: limit the concurrency for marshaling time series before sending them to remote storage There is no sense in running more than GOMAXPROCS concurrent marshalers, since they are CPU-bound. More concurrent marshalers do not increase the marshaling bandwidth, but they may result in more RAM usage.	2024-01-30 12:20:27 +02:00
Aliaksandr Valialkin	61562cdee9	lib/storage: keep (date, metricID) entries only for the last two dates Entries for the previous dates is usually not used, so there is little sense in keeping them in memory. This should reduce the size of storage/date_metricID cache, which can be monitored via vm_cache_entries{type="storage/date_metricID"} metric.	2024-01-29 18:44:27 +01:00
Aliaksandr Valialkin	36be93f53c	docs/keyConcepts.md: clarify the information about which data is returned by instant and range queries Do not use `raw samples` term there, since it adds more confusion than clarity: the `raw samples` refers to real samples stored in the database, while neither range nor instant queries do not return raw samples - they both return calculated samples at the given timestamps. This is a follow-up for `b5978ed8f9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5710 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5708	2024-01-29 18:22:27 +01:00
Fred Navruzov	ff67f4d3fb	- fix link/version of helm chart in update request (#5716 )	2024-01-29 18:22:11 +01:00
Artem Navoiev	00db646851	docs: add backward compaitble redicrt for url examples page Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-29 18:21:03 +01:00
Fred Navruzov	8043cf35d8	- update versions to 1.9.2 (#5714 ) - update guide asset urls to flat	2024-01-29 18:20:38 +01:00
hagen1778	131afea8ab	docs: specify results of Instant and Range queries Mention explicitly what are value and timestamp field in returned results from Instant and Range queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5710 https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5708 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 18:20:02 +01:00
Roman Khavronenko	a3e198588f	vmalert: set `ActiveAt` to evaluation timestamp in `newAlert` fn (#5657 ) The change fixes flaky test `TestAlertingRule_Exec` which has dependency on the actual timestamps, which resulted into inaccurate test states: https://github.com/VictoriaMetrics/VictoriaMetrics/actions/runs/7608452967/job/20717699688 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 17:30:14 +01:00
hagen1778	2ff94b2bfa	lib/streamaggr: fix incorrect err message for min `interval` value Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 17:27:23 +01:00
hagen1778	5aa0f77d8c	dashboards: specify where to see details about dropped labels Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 17:23:38 +01:00
Fred Navruzov	790ee9af6d	update guide and changelog to 1.9.1 (#5706 )	2024-01-29 17:23:06 +01:00
hagen1778	216f268c1a	docs: follow-up after `491287ed15` * port un-synced changed from docs/readme to readme * consistently use `sh` instead of `console` highlight, as it looks like a more appropriate syntax highlight * consistently use `sh` instead of `bash`, as it is shorter * consistently use `yaml` instead of `yml` See syntax codes here https://gohugo.io/content-management/syntax-highlighting/ Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 17:06:26 +01:00
hagen1778	3baeface89	docs: follow-up after `491287ed15` `491287ed15` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-29 16:59:13 +01:00
Artem Navoiev	0d210308c3	docs: remove witdh from images, remove <p>, remove <div> (#5705 ) * docs: remove witdh from images, remove <p>, remove <div> Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * docs: remove <div> clarify language in code blocks Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-29 16:50:51 +01:00
Daria Karavaieva	469bbbb2fa	version 1.9.1 update, dashboard viz flag (#5704 )	2024-01-29 16:48:55 +01:00
Aliaksandr Valialkin	f5559c038c	lib/storage: do not check the limit for -search.maxUniqueTimeseries when performing /api/v1/labels and /api/v1/label/.../values requests This limit has little sense for these APIs, since: - Thses APIs frequently result in scanning of all the time series on the given time range. For example, if extra_filters={datacenter="some_dc"} . - Users expect these APIs shouldn't hit the -search.maxUniqueTimeseries limit, which is intended for limiting resource usage at /api/v1/query and /api/v1/query_range requests. Also limit the concurrency for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests in order to limit the maximum memory usage and CPU usage for these API. This limit shouldn't affect typical use cases for these APIs: - Grafana dashboard load when dashboard labels should be loaded - Auto-suggestion list load when editing the query in Grafana or vmui Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055	2024-01-29 16:44:46 +01:00
Aliaksandr Valialkin	412f872597	lib/decimal: follow-up for `e6bad5174f` - Add a benchmark for CalbirateAndScale. - Reduce the decimal multipliers table size from 256Kb to 192bytes. - Use more clear naming for variables. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5672	2024-01-27 00:08:32 +01:00
Fuchun Zhang	e6bad5174f	Optimize the performance of data merge: decimal.CalibrateScale() (#5672 ) * Optimize the performance of data merge: decimal.CalibrateScale() from 49633 ns/op to 9146 ns/op * Optimize the performance of data merge: decimal.CalibrateScale()	2024-01-27 00:05:04 +01:00
Aliaksandr Valialkin	66a1541ce6	vendor: run `make vendor-update`	2024-01-26 22:59:59 +01:00
Aliaksandr Valialkin	2fba741af2	app/vmui: run `make vmui-update` after `a7b11eff7c`	2024-01-26 22:53:52 +01:00
Hui Wang	f579adf05f	add inserting comma inside value instruction to flag description (#5666 )	2024-01-26 22:47:33 +01:00
Roman Khavronenko	9e9f170fe7	lib/streamaggr: skip unfinished aggregation state on shutdown by default (#5689 ) Sending unfinished aggregate states tend to produce unexpected anomalies with lower values than expected. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-26 22:45:45 +01:00
Roman Khavronenko	562edb72ea	app/vmalert: fix data race during hot-config reload (#5698 ) * app/vmalert: fix data race during hot-config reload During hot-reload, the logic evokes the group update and rules evaluation interruption simultaneously. Falsely assuming that interruption happens before the update. However, it could happen that group will be updated first and only after the rules evaluation will be cancelled. Which will result in permanent interruption for all rules within the group. The fix caches the cancel context function into local variable first. And only after performs the group update. With cached cancel function we can safely call it without worrying that we cancel the evaluation for already updated group. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Revert "app/vmalert: fix data race during hot-config reload" This reverts commit `a4bb7e8932`. * app/vmalert: fix data race during hot-config reload During hot-reload, the logic evokes the group update and rules evaluation interruption simultaneously. Falsely assuming that interruption happens before the update. However, it could happen that group will be updated first and only after the rules evaluation will be cancelled. Which will result in permanent interruption for all rules within the group. The fix cancels the evaulation context before applying the update, making sure that the context will be cancelled for old group always. Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-26 22:43:02 +01:00
Yury Molodov	551f48466c	vmui: fix `Enter` key in query field (#5667 ) (#5681 )	2024-01-26 22:38:51 +01:00
Artem Navoiev	d42908133c	docs: remove <p> for imanges (#5702 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 22:34:40 +01:00
Artem Navoiev	36fa314161	remove all <div> as far they obsolete and can break markdown (#5701 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 22:32:54 +01:00
Aliaksandr Valialkin	2b0123058a	docs: update -help output after `bb7a419cc3`	2024-01-26 22:29:22 +01:00
Aliaksandr Valialkin	7a8b92b590	lib/{mergeset,storage}: make background merge more responsive and scalable - Maintain a separate worker pool per each part type (in-memory, file, big and small). Previously a shared pool was used for merging all the part types. A single merge worker could merge parts with mixed types at once. For example, it could merge simultaneously an in-memory part plus a big file part. Such a merge could take hours for big file part. During the duration of this merge the in-memory part was pinned in memory and couldn't be persisted to disk under the configured -inmemoryDataFlushInterval . Another common issue, which could happen when parts with mixed types are merged, is uncontrolled growth of in-memory parts or small parts when all the merge workers were busy with merging big files. Such growth could lead to significant performance degradataion for queries, since every query needs to check ever growing list of parts. This could also slow down the registration of new time series, since VictoriaMetrics searches for the internal series_id in the indexdb for every new time series. The third issue is graceful shutdown duration, which could be very long when a background merge is running on in-memory parts plus big file parts. This merge couldn't be interrupted, since it merges in-memory parts. A separate pool of merge workers per every part type elegantly resolves both issues: - In-memory parts are merged to file-based parts in a timely manner, since the maximum size of in-memory parts is limited. - Long-running merges for big parts do not block merges for in-memory parts and small parts. - Graceful shutdown duration is now limited by the time needed for flushing in-memory parts to files. Merging for file parts is instantly canceled on graceful shutdown now. - Deprecate -smallMergeConcurrency command-line flag, since the new background merge algorithm should automatically self-tune according to the number of available CPU cores. - Deprecate -finalMergeDelay command-line flag, since it wasn't working correctly. It is better to run forced merge when needed - https://docs.victoriametrics.com/#forced-merge - Tune the number of shards for pending rows and items before the data goes to in-memory parts and becomes visible for search. This improves the maximum data ingestion rate and the maximum rate for registration of new time series. This should reduce the duration of data ingestion slowdown in VictoriaMetrics cluster on e.g. re-routing events, when some of vmstorage nodes become temporarily unavailable. - Prevent from possible "sync: WaitGroup misuse" panic on graceful shutdown. This is a follow-up for `fa566c68a6` . Thanks @misutoth to for the inspiration at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2024-01-26 22:19:52 +01:00
Artem Navoiev	9e0416c666	docs: delete docs/provision_datasources.png as we support webp Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:36:19 +01:00
Github Actions	41c48a4c59	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@ef5cfe6 (#5700 )	2024-01-26 21:35:53 +01:00
Aliaksandr Valialkin	c067f3f288	lib/mergeset: remove inmemoryBlock pooling, since it wasn't effecitve This should reduce memory usage a bit when new time series are ingested at high rate (aka high churn rate)	2024-01-26 21:34:22 +01:00
Aliaksandr Valialkin	d8c82b6421	app/vmselect/netstorage: initialize tmpBlocksFileWrapper at goroutine, which continues using it This may improve CPU cache locality	2024-01-26 21:29:30 +01:00
Aliaksandr Valialkin	230ef43a32	lib/logstorage: make sure that WaitGroup.Add isnt called after stopCh is closed and WaitGroup.Wait is called This protects from rare panic, which may occur during graceful shutdown of VictoriaLogs	2024-01-26 21:18:07 +01:00
Aliaksandr Valialkin	9e70d9ab47	docs/Makefile: mention that the Makefile rules must be run from VictoriaMetrics repository root	2024-01-26 21:11:14 +01:00
Aliaksandr Valialkin	7c7bfa27ac	app/vmauth: return 503 service unavailable status code when the backend returns response with unsupported status code, but the request cannot be re-tried. While at it, properly close response body. This should prevent from possible http keep-alive connection leak to backends because of unclosed response bodies. This is a follow-up for `3c0aa14b5b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5688	2024-01-26 21:10:57 +01:00
Artem Navoiev	2b870b1116	docs: fix key concepts image and links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:10:36 +01:00
Artem Navoiev	bb5a0719a5	docs: change [image] to img as far we support it in release guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:09:22 +01:00
Artem Navoiev	89fbefefdb	docs: remoev vmanomaly as far we have dedicated section with alredy exists redirects Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:08:54 +01:00
Artem Navoiev	6ed9a05a08	docs: vmanomaly fix images Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:08:28 +01:00
Daria Karavaieva	b042982339	Vmanomaly Guide dashboard provisioning (#5679 ) * dashboard provisioning * delete dashboard filter, new query * dashboard screens, guide fixes	2024-01-26 21:07:44 +01:00
Artem Navoiev	aee3e51315	docs: remove raw and endraw tags as they are not needed for the new v… (#5696 ) * docs: remove raw and endraw tags as they are not needed for the new version of site Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * revert formating in vmaler Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-26 21:06:10 +01:00
Github Actions	cd287b2e4c	Automatic update operator docs from VictoriaMetrics/operator@0628def (#5694 )	2024-01-26 21:05:13 +01:00
Github Actions	b8a4a78fef	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@c644bec (#5691 )	2024-01-26 20:52:47 +01:00
Roman Khavronenko	a2f83115ae	app/vmalert: autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0` (#5680 ) * app/vmalert: autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0` Previously, `ALERTS_FOR_STATE` was generated only for alerts with `for > 0`. This behavior differs from Prometheus behavior - it generates ALERTS_FOR_STATE time series for alerting rules with `for: 0` as well. Such time series can be useful for tracking the moment when alerting rule became active. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5648 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3056 Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmalert: support ALERTS_FOR_STATE in `replay` mode Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-26 20:51:50 +01:00
Github Actions	362e52f880	Automatic update operator docs from VictoriaMetrics/operator@e75a096 (#5690 )	2024-01-26 20:50:57 +01:00
hagen1778	d76e338505	docs: simplify instructions for spinning up docker env Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-26 20:50:31 +01:00
Github Actions	8fa6c42396	Automatic update operator docs from VictoriaMetrics/operator@f6b9c08 (#5676 )	2024-01-26 20:50:00 +01:00
Alexander Marshalov	ef4bb36d99	vmauth: fix `vmauth_user_request_backend_errors_total` metric calc logic for use case when only one backend is available - if we get an error from the retry_status_codes list, but cannot execute retry, we increment vmauth_user_request_backend_errors_total as well (#5688 )	2024-01-26 20:49:18 +01:00
hagen1778	bd7ebb41b2	docs: fix the issue link Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-26 20:49:01 +01:00
Alexander Marshalov	1f858eb417	Follow up after https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5685 (#5686 )	2024-01-26 20:08:15 +01:00
Alexander Marshalov	711ace0582	Make a makefile work on a MacOS and Linux (#5685 )	2024-01-26 20:07:20 +01:00
Aliaksandr Valialkin	0715f1efcd	lib/storage: rename AssistedMerges to AssistedMergesCount in order to make these field names less misleading These fields are counters, not gauges, so adding Count suffix to them makes easier to understand this while reading the code	2024-01-25 10:21:13 +02:00
Alexander Marshalov	14712e3b99	vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached. (#5682 )	2024-01-25 10:21:09 +02:00
Aliaksandr Valialkin	1cdef56d84	lib/mergeset: start assisted merge for file parts only if the number of file parts is bigger than maxFileParts The maxFileParts usage has been accidentally removed in `fa566c68a6` While at it, add Count suffix to *AssistedMerges counter names in order to make them less misleading. Previously their names were falsely suggesting that these are gauges, which show the number of concurrently executed assisted merges.	2024-01-24 15:10:48 +02:00
Aliaksandr Valialkin	b8c7f0d3bc	lib/promscrape/discovery/kubernetes: typo fix in the comment for ContainerStateTerminated struct This is a follow-up for `ef12598ad4`	2024-01-24 15:10:47 +02:00
Aliaksandr Valialkin	1e364c992d	lib/promscrape/discovery/kubernetes: do not generate targets for already terminated pods and containers Already terminated pods and containers cannot be scraped and will never resurrect, so there is zero sense in creating scrape targets for them.	2024-01-24 14:58:51 +02:00
Aliaksandr Valialkin	0dca3c4025	app/{vmselect,vmstorage}: return compression of the data passed from vmstorage to vmselect This reverts `cd4f641d32` , since it has been appeared that the disabled compression for vmstorage->vmselect data increase network bandwidth usage by more than 10x on typical production workloads, while it decreases CPU usage at vmstorage by up to 10% and improves query latency by up to 10%. The 10x increase in network usage is too high price for 10% improvements on query latency and vmstorage CPU usage. This may result in network bandwidth bottlenecks, which can reduce the overall performance and stability of VictoriaMetrics cluster. That's why return back the vmstorage->vmselect data compression by default. The vmstorage->vmselect compression can be disabled by passing -rpc.disableCompression command-line flag to vmstorage. The vmselect->vmselect compression in multi-level cluster setup can be disabled by passing -clusternative.disableCompression command-line flag.	2024-01-24 13:37:05 +02:00
Aliaksandr Valialkin	e6e5b97e1e	lib/streamaggr: expand `%{ENV}` placeholders in stream aggregation configs	2024-01-24 12:31:42 +02:00
Aliaksandr Valialkin	12698b9136	lib/mergeset: really limit the number of in-memory parts to 15 It has been appeared that the registration of new time series slows down linearly with the number of indexdb parts, since VictoriaMetrics needs to check every indexdb part when it searches for TSID by newly ingested metric name. The number of in-memory parts grows when new time series are registered at high rate. The number of in-memory parts grows faster on systems with big number of CPU cores, because the mergeset maintains per-CPU buffers with newly added entries for the indexdb, and every such entry is transformed eventually into a separate in-memory part. The solution has been suggested in https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 by @misutoth - to limit the number of in-memory parts with buffered channel. This solution is implemented in this commit. Additionally, this commit merges per-CPU parts into a single part before adding it to the list of in-memory parts. This reduces CPU load when searching for TSID by newly ingested metric name. The https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 recommends setting the limit on the number of in-memory parts to 100, but my internal testing shows that much lower limit 15 works with the same efficiency on a system with 16 CPU cores while reducing memory usage for `indexdb/dataBlocks` cache by up to 50%. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190	2024-01-24 03:41:19 +02:00
Aliaksandr Valialkin	5205f1c6a6	docs/Cluster-VictoriaMetrics.md: document that `vmstorage` doesnt compress data it sends to `vmselect` by default This is a follow-up for `cd4f641d32`	2024-01-23 23:21:31 +02:00
Aliaksandr Valialkin	8dd73574ca	lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint*	2024-01-23 22:50:11 +02:00
Aliaksandr Valialkin	5a97668ad6	lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow	2024-01-23 18:39:28 +02:00
Aliaksandr Valialkin	be320c81bc	app/vminsert/clusternative: explain why lower-level vminsert doesnt compress responses to upper-level vminsert	2024-01-23 18:14:19 +02:00
Aliaksandr Valialkin	3199558da9	lib/{storage,mergeset}: reduce the maxium compression level for the stored data This reduces CPU usage a bit, while doesn't increase resulting file sizes according to synthetic tests.	2024-01-23 17:47:40 +02:00
Github Actions	4ccf3f41c6	Automatic update operator docs from VictoriaMetrics/operator@1470569 (#5668 )	2024-01-23 17:47:36 +02:00
Aliaksandr Valialkin	68d76b1436	lib/storage: compress metricIDs, which match the given filters, before storing them in tagFiltersToMetricIDsCache This allows reducing the indexdb/tagFiltersToMetricIDs cache size by 8 on average. The cache size can be checked via vm_cache_size_bytes{type="indexdb/tagFiltersToMetricIDs"} metric exposed at /metrics page.	2024-01-23 16:13:25 +02:00
Aliaksandr Valialkin	9b3217db61	lib/storage: do not sort metricIDs passed to Storage.prefetchMetricNames, since the caller is responsible for the sorting	2024-01-23 16:13:19 +02:00
Aliaksandr Valialkin	7ed7eb95b4	lib/filestream: do not measure read / write duration from / to in-memory buffers Measuring read / write duration from / to in-memory buffers has little sense, since it will be always fast. It is better to measure read / write duration from / to real files at vm_filestream_write_duration_seconds_total and vm_filestream_read_duration_seconds_total metrics. This also reduces overhead on time.Now() and Histogram.UpdateDuration() calls per each filestream.Reader.Read() and filestream.Writer.Write() call when the data is read / written from / to in-memory buffers. This is a follow-up for `2f63dec2e3`	2024-01-23 14:53:35 +02:00
Aliaksandr Valialkin	cfc1193d15	app/vmselect/netstorage: limit the maximum brsPool size to 32Kb at ProcessSearchQuery() This avoids slow path in Go runtime for allocating objects bigger than 32Kb - see `704401ffa0/src/runtime/malloc.go (L11)` This also reduces memory usage a bit for vmselect and single-node VictoriaMetrics after the commit `5dd37ad836` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527	2024-01-23 14:12:27 +02:00
Aliaksandr Valialkin	fe4ea30a79	app/vmselect/netstorage: limit the size of metricNamesBuf to 32Kb in order to avoid slow path at Go runtime for allocating a byte slice of bigger size See `704401ffa0/src/runtime/malloc.go (L11)` This also reduces the average memory usage a bit for vmselect and single-node VictoriaMetrics after the commit `508c608062` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527	2024-01-23 13:50:59 +02:00
Aliaksandr Valialkin	47cb79198e	docs/vmagent.md: clarify how `-promscrape.seriesLimitPerTarget` command-line flag, `series_limit` config option and `__series_limit__` label interact with each other This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5663 See also `89e3c70ccd`	2024-01-23 13:15:45 +02:00
Roman Khavronenko	8461add541	lib/promscrape: respect `0` value for `series_limit` param (#5663 ) * lib/promscrape: respect `0` value for `series_limit` param Respect `0` value for `series_limit` param in `scrape_config` even if global limit was set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value will be ignored in favor of `-promscrape.seriesLimitPerTarget`. This behavior aligns with possibility to override `series_limit` value via relabeling with `__series_limit__` label. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update docs/CHANGELOG.md --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-23 13:09:36 +02:00
Aliaksandr Valialkin	c2927053ee	lib/mergeset: make sure that the first and the last items are in the original range after prepareBlock() Previously the checks were to strict by requiring to leave the same first and last items by prepareBlock() Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5655	2024-01-23 12:59:04 +02:00
Fred Navruzov	7b1325c3a3	- fix 404 errors after page remaning (#5664 ) - slight text fixes	2024-01-23 12:59:01 +02:00
Aliaksandr Valialkin	1880c656c1	app/vmselect/vmui: run `make vmui-update` in order to sync recent changes in app/vmui	2024-01-23 04:31:57 +02:00
Yury Molodov	1db2b991b7	vmui: query report (#5497 ) * vmui: add query analyzer page * vmui: fix tabs for query analyzer * vmui: add help to export query * vmui: add time params to query analyzer * docs/vmui: add query analyzer * vmui: fix validation JSON form --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-23 04:26:04 +02:00
Yury Molodov	3a26e4d6ec	vmui: add flag for default timezone setting (#5611 ) * vmui: add flag for default timezone setting #5375 * vmui: validate timezone before client return * Update app/vmselect/vmui.go --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-23 04:15:14 +02:00
Yury Molodov	574d69775e	vmui: fix cache autocomplete (#5591 ) * vmui: fix the logic of closing the popper #5470 * vmui: fix the logic of caching autocomplete results #5472 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-23 04:06:39 +02:00
Aliaksandr Valialkin	389159767d	lib/mergeset: skip comparison for every item in the block during merge if the last item in the block is smaller than the first item in the next block Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5651	2024-01-23 03:16:30 +02:00
Daria Karavaieva	7ad16c8f17	add 1.8.0 notes to changelog (#5616 ) * add 1.8.0 notes to changelog * added release date * MAD internal link * monitoring health deprecation	2024-01-23 02:31:21 +02:00
Jaskeerat Singh Randhawa	291802ea84	custom-resources: fix link text for alertmanager (#5660 )	2024-01-23 02:30:51 +02:00
Aliaksandr Valialkin	e20cbfcbc3	app/vmselect/promql: remove superflouos memory allocations at aggrPrepareSeries() While at it, also remove unneeded map lookup	2024-01-23 02:29:14 +02:00
Aliaksandr Valialkin	cd4f641d32	app/{vmstorage,vmselect}: disable vmstorage->vmselect RPC compression by default in order to improve query performance	2024-01-23 02:29:13 +02:00
Aliaksandr Valialkin	b7cc1af3eb	app/vmselect/promql/aggr_incremental.go: eliminate unnecessary memory allocation in incrementalAggrFuncContext.updateTimeseries	2024-01-23 02:29:13 +02:00
Aliaksandr Valialkin	953b96ced2	app/vmselect/netstorage: remove tswPool, since it isnt efficient	2024-01-23 02:29:13 +02:00
Aliaksandr Valialkin	68a59bfabd	app/vmselect/netstorage: avoid metricName->blockRef lookup when processing multiple blocks for the same time series This saves a few CPU cycles for common case	2024-01-23 02:29:12 +02:00
Aliaksandr Valialkin	f8a9ef8cbd	app/vmselect/netstorage: group per-vmstorage fields at tmpBlocksFileWrapperShard This improves code readability a bit	2024-01-23 02:29:12 +02:00
Aliaksandr Valialkin	d52b121222	app/vmselect/netstorage: use []blockRef from blockRefPool in order to reduce memory allocations	2024-01-23 02:29:11 +02:00
Aliaksandr Valialkin	5b05224eb9	app/vmselect/netstorage: substitute pointer to blockRefs by brssPool index at the metricName->blockRefs map This should reduce the pressure on Go GC, since it will see lower number of pointers. This change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527	2024-01-23 02:29:11 +02:00
Aliaksandr Valialkin	b289f15f02	app/vmselect/netstorage: reduce the number of allocations for blockRefs objects in ProcessSearchQuery() This should reduce pressure on Go GC at vmselect The change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527	2024-01-23 02:29:11 +02:00
Aliaksandr Valialkin	2ab9a75cca	app/vmselect/netstorage: reduce the number of memory allocations in ProcessSearchQuery() by storing all the metric names in a single byte slice This reduces the number of memory allocations at the cost of possible memory usage increase, since now different metric name strings may hold references to the previous byte slice. This is good tradeoff, since ProcessSearchQuery is called in vmselect, and vmselect isn't usually limited by memory. This change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527	2024-01-23 02:29:10 +02:00
Zakhar Bessarab	60ef978ffc	lib/storage: print tenant ID in log when discarding or truncating labels (#5658 ) Previously, it was not possible to determine which tenant sends metrics with excessive amount of labels of label values. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-23 02:27:59 +02:00
hagen1778	22a8664f51	deployment/docker: fix typo in commands example Follow up after `38b2a5bc44` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 18:44:52 +02:00
hagen1778	6ef6b83b33	dashboards: reflect dashboard rename in copy script This is a follow-up for `ff33e60a3d` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 18:44:18 +02:00
hagen1778	644d43620f	deployment/docker: follow-up `38b2a5bc44` * Simplify folder structure * mention datasource in README Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 18:43:54 +02:00
Dmytro Kozlov	69e59ac9b7	deployment/docker: add grafana datasource to the docker-compose files (#5363 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3920 https://github.com/VictoriaMetrics/grafana-datasource/issues/113	2024-01-22 18:43:16 +02:00
hagen1778	8138499439	app/vmctl/backoff: fix flaky test The change removes artificial delay before returning error, which sometimes caused less retry events than expected. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 18:42:05 +02:00
hagen1778	ede466be56	docs: fix Grafana link example for vmalert Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 18:41:38 +02:00
Aliaksandr Valialkin	d52fd73f18	all: add up to 10% random jitter to the interval between periodic tasks performed by various components This should smooth CPU and RAM usage spikes related to these periodic tasks, by reducing the probability that multiple concurrent periodic tasks are performed at the same time.	2024-01-22 18:39:16 +02:00
Aliaksandr Valialkin	64e615e6cc	lib/storage: reduce the contention on dateMetricIDCache mutex when new time series are registered at high rate The dateMetricIDCache puts recently registered (date, metricID) entries into mutable cache protected by the mutex. The dateMetricIDCache.Has() checks for the entry in the mutable cache when it isn't found in the immutable cache. Access to the mutable cache is protected by the mutex. This means this access is slow on systems with many CPU cores. The mutabe cache was merged into immutable cache every 10 seconds in order to avoid slow access to mutable cache. This means that ingestion of new time series to VictoriaMetrics could result in significant slowdown for up to 10 seconds because of bottleneck at the mutex. Fix this by merging the mutable cache into immutable cache after len(cacheItems) / 2 cache hits under the mutex, e.g. when the entry is found in the mutable cache. This should automatically adjust intervals between merges depending on the addition rate for new time series (aka churn rate): - The interval will be much smaller than 10 seconds under high churn rate. This should reduce the mutex contention for mutable cache. - The interval will be bigger than 10 seconds under low churn rate. This should reduce the uneeded work on merging of mutable cache into immutable cache.	2024-01-22 18:14:30 +02:00
dependabot[bot]	9c153d0710	build(deps): bump github/codeql-action from 2 to 3 (#5462 ) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-01-22 01:49:40 +02:00
Aliaksandr Valialkin	c6f6f094c5	Revert "lib/promscrape: do not store last scrape response when stale markers … (#5577 )" This reverts commit `cfec258803`. Reason for revert: the original code already doesn't store the last scrape response when stale markers are disabled. The scrapeWork.areIdenticalSeries() function always returns true is stale markers are disabled. This prevents from storing the last response at scrapeWork.processScrapedData(). It looks like the reverted commit could also return back the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5577	2024-01-22 01:46:12 +02:00
Aliaksandr Valialkin	3230525c36	docs: use persistent links to Grafana dashboards These links do not depend on the dashboard name, so they do not break after the renaming of the dashboard. This is a follow-up for `ff33e60a3d`	2024-01-22 01:45:42 +02:00
Aliaksandr Valialkin	d4a1a28543	app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery() This is a follow-up for `cf03e11d89` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5630	2024-01-22 01:39:27 +02:00
Artem Navoiev	fbbd5ab1e7	docs vmanomaly fix anchor Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 01:36:22 +02:00
Hui Wang	49fa92c1d0	lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice (#5557 ) * lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice Previously the groupWatcher could be mistakenly stopped when requests for pod or services resources take too long. * remove mislead comment * docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640 * wip * lib/promscrape/kubernetes: prevent from stopping groupWatcher when there are in-flight apiWatcher.mustStart() calls groupWatcher is stopped if it has zero registered apiWatchers during 14 seconds. But such a groupWatcher can be still in use if apiWatcher for `role: endpoints` or `role: endpointslice` is being registered and the discovery of the associated `pod` and/or `service` objects takes longer than 14 seconds - see the beginning of groupWatcher.startWatchersForRole() function for details. Track the number of in-flight calls to apiWatcher.mustStart() and prevent from stopping the associated groupWatcher if the number of in-flight calls is non-zero. P.S. postponing the discovery of `pod` and/or `service` objects associated with `endpoints` or `endpointslice` roles isn't the best solution, since it slows down initial discovery of `endpoints` and `endpointslice` targets. * typo fix --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-22 01:33:17 +02:00
Aliaksandr Valialkin	885ee160c2	all: allow dynamically reading *AuthKey flag values from files and urls Examples: 1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath 2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath 3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url The flag value is automatically updated when the file contents changes.	2024-01-22 01:23:23 +02:00
Fred Navruzov	b76c489e57	- fix 404 links after renaming (#5653 ) - improve wording on diagram - swap enterprise/about chapters for page clarity	2024-01-22 01:14:54 +02:00
Fred Navruzov	11c5b5d3ab	docs: vmanomaly - add component interaction diagram (#5652 ) * add interaction diagram for vmanomaly components * small docs fixes * resolve suggestions	2024-01-22 01:14:16 +02:00
Aliaksandr Valialkin	5f5fcab217	all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time This allows avoiding slow inter-CPU synchornization induced by atomic.CompareAndSwap*	2024-01-22 01:13:41 +02:00
Aliaksandr Valialkin	6d91d10cbd	docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640	2024-01-22 01:13:19 +02:00
Aliaksandr Valialkin	be5faef552	lib/promscrape: code cleanup: send stale markers immediately after generating automatic metrics This cleanup has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557/files#diff-6b205cf6637d7b65a5c45d9417d08822d4efad94227268cb196f61aa2a0fc0f7	2024-01-22 01:12:56 +02:00
Aliaksandr Valialkin	e15f07d989	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-22 01:11:59 +02:00
Aliaksandr Valialkin	2f94bef59c	lib/storage/partition.go: remove misleading comment, which falsely states that inmemoryParts isn't visible to search Thanks to @satjd for raising attention to this comment at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5410	2024-01-22 01:11:36 +02:00
Nikolay	73c51072e6	app/vmauth: adds metric_labels and backend_errors counter (#5585 ) * app/vmauth: adds metric_labels and backend_errors counter it must improve observability for user requests with new metric - per user backend errors counter. it's needed to calculate requests fail rate to the configured backends. metric_labels configuration allows to perform additional aggregations on top of multiple users from configuration section. It could be multiple clients or clients with separate read/write tokens https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-22 01:09:51 +02:00
Yury Molodov	0582ec5c8c	vmui: add autofocus to input for desktop version #5479 (#5592 )	2024-01-22 01:09:27 +02:00
Aliaksandr Valialkin	2c7c812a9d	lib/promscrape/discovery/kubernetes: add -promscrape.kubernetes.attachNodeMetadataAll command-line flag This flag allows setting attach_metadata.node=true for all the kubernetes_sd_configs defined at -promscrape.config Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640 Thanks to wasim-nihal for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593	2024-01-22 01:08:52 +02:00
Hui Wang	e086ef16da	app/vmselect/promql: properly handle possible negative results caused… (#5608 ) * app/vmselect/promql: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase() * fix test	2024-01-22 01:04:50 +02:00
Nikolay	d79a7c36b0	app/vmselect/netstorage (#5649 ) * app/vmselect/netstorage correctly handle errGlobal set * wip Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-22 01:01:38 +02:00
Nikolay	e196c61e36	app/vmselect: abort streaming connections for vmselect (#5650 ) * app/vmselect: abort streaming connections for vmselect due to streaming nature of export APIs, curl and simmilr tools cannot detect errors that happened after http.Header with status 200 was written to it. This PR tracks if body write was already started and closes connection. It allows client to detect not expected chunk sequence and return error to the caller. Mostly it affects vmselect at cluster version https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645 * wip Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5650 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-22 00:54:32 +02:00
Aliaksandr Valialkin	c05982bfa7	lib/promscrape/discovery/hetzner: follow-up after `03a97dc678` - docs/sd_configs.md: moved hetzner_sd_configs docs to the correct place according to alphabetical order of SD names, document missing __meta_hetzner_role label. - lib/promscrape/config.go: added missing MustStop() call for Hetzner SD, and moved the code to the correct place according to alphabetical order of SD names. - lib/promscrape/discovery/hetzner: properly handle pagination for hloud API responses, populate missing __meta_hetzner_role label like Prometheus does. - Properly populate __meta_hetzner_public_ipv6_network label like Prometheus does. - Remove unused SDConfig.Token. - Remove "omitempty" annotation from SDConfig.Role field, since this field is mandatory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154	2024-01-22 00:53:23 +02:00
Artem Navoiev	1e14c3177b	vmanomly use proper title for overiview doc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:52:56 +02:00
Hui Wang	66eb013b54	lib/promscrape: do not store last scrape response when stale markers … (#5577 ) * lib/promscrape: do not store last scrape response when stale markers are disabled * update changelog	2024-01-22 00:52:25 +02:00
Artem Navoiev	75920ab3b1	docs add docs titles Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:28:54 +02:00
Artem Navoiev	c8fc813e00	vmanonamly docs fix sorting for jekill as far it doesnt support of sorting the folders Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:28:17 +02:00
Artem Navoiev	f3b4ca6a77	vmanomaly remove unused pages from menu Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:27:49 +02:00
Artem Navoiev	7eff777467	vmanomly specify the right menu parent for overview page Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:27:26 +02:00
Artem Navoiev	0ec4134ab6	docs vmanmoly fix sorting Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:26:59 +02:00
Artem Navoiev	b5363ffab2	Move vmanomaly page to its own section (#5646 ) * docs: move vmanomaly overview page to its section Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * add alias for backward compatibility Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix links Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * change title Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:26:37 +02:00
Roman Khavronenko	fe4934f0ec	app/vmui: send `step` param for instant queries (#5639 ) The change reverts https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896 due to reasons explained in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896#issuecomment-1896704401 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 00:25:31 +02:00
Fred Navruzov	3b8f4714c3	docs: vmanomaly slight improvements (#5637 ) * - better messaging - update links to dockerhub in guides - added anomaly_score to FAQ - improve model section (sort + use cases) - slight refactor of a guide * rename guide & change refs * change wording in installation options * - update remaining text reference - add cross-link to component sections in guide * add docs/.jekyll-metadata to .gitignore	2024-01-22 00:24:39 +02:00
hagen1778	a99d26633b	docs: remove slug from Grafana dashboard URLs Each Grafana dashboard has unique ID which can be used to fetch the dashboard from grafana.com: https://grafana.com/grafana/dashboards/11176 The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/ But using slug implies that any change to dashboard name will break the link. So it is better to just use ID, so the dashboard URL will never break. This is follow-up for `ff33e60a3d` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-22 00:23:30 +02:00
Artem Navoiev	70743c7014	fix link for grafana dashbaord for single node after its renaming Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:21:57 +02:00
Artem Navoiev	79b308fa54	docs: vmanomaly update vmanomaly + vmalert guide (#5636 ) * docs: vmanomaly update vmanomaly + vmalert guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * docs: vmanomaly update vmanomaly + vmalert guide. Update docker compose and monitoring section Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * typos and fixes Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-22 00:19:15 +02:00
Artem Navoiev	8154d16420	docs: changelog fix the link to cluster Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-21 23:54:08 +02:00
Artem Navoiev	b1fe59df8b	vmanomaly docs fix broken relative links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-21 23:53:39 +02:00
Artem Navoiev	547b62a807	docs/anomaly-detection/components/models.md add sort:1 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-21 23:53:10 +02:00
Artem Navoiev	7c62010d04	vmanonaly docs add .html for the section document models Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-21 23:52:41 +02:00
Artem Navoiev	b66b059231	vmanomaly docs simplify the strcuture (#5634 ) * vmanomaly docs simplify the strcuture Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix links Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-21 23:52:07 +02:00
hagen1778	07a55a484a	docs/setup-size: rm tolerable churn rate % It is likely this value was borrowed from panel `Slow inserts` panel from Grafana dasbhoard for VM single/cluster installations. This is a mistake. There is no such thing as "tolerable churn rate", as tolerancy depends on the amount of allocated resources. Although, it is unclear what is meant by 5%. If it refers to 5% of new time series per second, then such churn rate is extremely high. It would mean that the avg life of a time series is 20s. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-21 23:48:24 +02:00
Roman Khavronenko	148e14b3f2	app/vmselect: properly calculate `start` param for queries with too big look-behind window (#5630 ) Properly determine time range search for instant queries with too big look-behind window like `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-21 23:47:09 +02:00
Aliaksandr Valialkin	41d6c8a7dd	lib/storage: do not prefetch metric names for small number of metricIDs This eliminates prefetchedMetricIDsLock lock contention for queries, which return less than 500 time series. This is a follow-up for `9d886a2eb0`	2024-01-17 13:50:01 +02:00
Aliaksandr Valialkin	ecb0a3d27d	docs/keyConcepts.md: typo fixes after `b7ffee2644` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555	2024-01-17 13:27:37 +02:00
Aliaksandr Valialkin	b7ffee2644	docs/keyConcepts.md: document /internal/force_flush handler Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555	2024-01-17 13:23:46 +02:00
Aliaksandr Valialkin	014bba6d8e	docs/CHANGELOG*: move changes for 2023 year to docs/CHANGELOG_2023.md	2024-01-17 13:12:58 +02:00
hagen1778	d4ae2cb5d0	docs/troubleshooting: mention query latency in unexpected query results See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-17 13:12:35 +02:00
Aliaksandr Valialkin	36e7689946	LICENSE: update the current year from 2023 to 2024	2024-01-17 01:48:12 +02:00
Aliaksandr Valialkin	a130b86533	docs/CHANGELOG.md: document v1.93.10 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.10	2024-01-17 01:45:22 +02:00
Aliaksandr Valialkin	09f23b0296	lib/promscrape: cosmetic changes after `3ac44baebe` - Rename mustLoadScrapeConfigFiles() to loadScrapeConfigFiles(), since now it may return error. - Split too long line with the error message into two lines in order to improve readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5508 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5560	2024-01-17 01:07:16 +02:00
Aliaksandr Valialkin	dc5066368a	docs/{vmbackup,vmbackupmanager}.md: clarify why storing backups to S3 Glacier can be time-consuming and expensive Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5614 This is a follow-up for `e14e3d9c8c`	2024-01-17 01:06:51 +02:00
hagen1778	a2d3fce05f	deployment/dashboards: change title `VictoriaMetrics` to `VictoriaMetrics - single-node` The new title should provide better understanding of this dashboard purpose. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-17 01:06:07 +02:00
Aliaksandr Valialkin	0b1c2f70a5	docs/CHANGELOG.md: document v1.87.13 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.13	2024-01-17 01:04:26 +02:00
Aliaksandr Valialkin	2b67944eb4	app/vmselect/graphite: properly handle -N index for the array of N items This is a follow-up for `70cd09e736` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5581	2024-01-17 00:16:37 +02:00
Aliaksandr Valialkin	1340ab1c07	docs/CHANGELOG.md: fix a link in the description of `70cd09e736` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5581	2024-01-17 00:16:17 +02:00
Aliaksandr Valialkin	b3823bc091	app/vmctl: disallow insecure https connections to vm-native-dst-addr and vm-native-src-addr by default It is better from security PoV to disallow insecure https connections to vm-native-dst-addr and vm-native-src-addr . This also maintains backwards compatibility with vmctl before the commit `828aca82e9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5595 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5606	2024-01-17 00:15:50 +02:00
Aliaksandr Valialkin	8d13b543ba	docs/Single-server-VictoriaMetrics.md: explain why staleness markers are treated as an ordinary values during de-duplication This is a follow-up for `d374595e31` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5587	2024-01-17 00:15:18 +02:00
Aliaksandr Valialkin	c819e85904	docs/vmagent.md: mention the minumum value for -remoteWrite.vmProtoCompressLevel Recommend using the default value for -remoteWrite.vmProtoCompressLevel This is a follow-up for `095d982976` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5575	2024-01-17 00:14:08 +02:00
Aliaksandr Valialkin	75c58ab306	lib/httputils: handle step=undefined query arg as an empty value This is needed for Grafana, which may send step=undefined when working with alerting rules and instant queries.	2024-01-17 00:13:04 +02:00
Yury Molodov	9588b9bd19	vmui/vmanomaly: add support models that produce only `anomaly_score` (#5594 ) * vmui/vmanomaly: add support models that produce only `anomaly_score` * vmui/vmanomaly: fix display legend * vmui/vmanomaly: update comment on anomaly threshold	2024-01-17 00:12:43 +02:00
Artem Navoiev	bdf4f0f1e2	docs: vmanomaly fix font matter Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-17 00:12:05 +02:00
Artem Navoiev	a1ae06cc04	docs: vmanomaly remove commented text Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-17 00:07:36 +02:00
Aliaksandr Valialkin	b7fcdb1985	deployment/docker: update Go builder from Go1.21.5 to Go1.21.6	2024-01-17 00:05:24 +02:00
Aliaksandr Valialkin	7c737846a6	vendor: run `make vendor-update`	2024-01-16 22:48:46 +02:00
Aliaksandr Valialkin	f673039e86	lib/storage: follow-up for `4b8088e377` - Clarify the bugfix description at docs/CHANGELOG.md - Simplify the code by accessing prefetchedMetricIDs struct under the lock instead of using lockless access to immutable struct. This shouldn't worsen code scalability too much on busy systems with many CPU cores, since the code executed under the lock is quite small and fast. This allows removing cloning of prefetchedMetricIDs struct every time new metric names are pre-fetched. This should reduce load on Go GC, since the cloning of uin64set.Set struct allocates many new objects.	2024-01-16 22:38:57 +02:00
Aliaksandr Valialkin	84f11a9e6d	app/vmselect/promql: simplify the code after `388d020b7c` Add a test, which verifies the correct sorting of float64 slices with NaNs. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509	2024-01-16 22:35:51 +02:00
Hui Wang	2f40ed3aac	exit vmagent if there is config syntax error in `scrape_config_files` when `-promscrape.config.strictParse=true` (#5560 )	2024-01-16 22:35:18 +02:00
hagen1778	343176438e	deployment/alerts: add `job` label to `DiskRunsOutOfSpace` alerting rule So it is easier to understand to which installation the triggered instance belongs. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 22:21:34 +02:00
Aliaksandr Valialkin	6ba2fd3312	app/vmselect/promql: follow-up for `ce4f26db02` - Document the bugfix at docs/CHANGELOG.md - Filter out NaN values before sorting as suggested at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509#discussion_r1447369218 - Revert unrelated changes in lib/filestream and lib/fs - Use simpler test at app/vmselect/promql/exec_test.go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506	2024-01-16 22:13:13 +02:00
Zongyang	cb37df5723	FIX bottomk doesn't return any data when there are no time range overlap between timeseries (#5509 ) * FIX sort order in bottomk * Add lessWithNaNsReversed for bottomk * Add ut for TopK * Move lt from loop * FIX lint * FIX lint * FIX lint * Mod log format --------- Co-authored-by: xiaozongyang <xiaozngyang@kanyun.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2024-01-16 22:12:49 +02:00
Aliaksandr Valialkin	015c0a4d1a	app/vmselect/promql: consistently sort results of `a or b` query Previously the order of results returned from `a or b` query could change with each request because the sorting for such query has been disabled in order to satisfy https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763 . This commit executes `a or b` query as `sortByMetricName(a) or sortByMetricName(b)`. This makes the order of returned time series consistent across requests, while maintaining the requirement from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763 , e.g. `b` results are consistently put after `a` results. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5393	2024-01-16 22:12:15 +02:00
Aliaksandr Valialkin	825cfdb5ef	app/vmstorage: expose proper types for storage metrics when -metrics.exposeMetadata command-line flag is set This is a follow-up for `326a77c697`	2024-01-16 22:03:31 +02:00
Aliaksandr Valialkin	724223fad4	lib/prompbmarshal: move WriteRequest proto definition to the correct place	2024-01-16 21:57:03 +02:00
Artem Navoiev	55a03ff454	docs: vmanomaly fix image size Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:56:25 +02:00
Artem Navoiev	83287bde1c	vmanomly: guide add diagramm Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:55:59 +02:00
Artem Navoiev	f3364bbd8b	docs: vmanomaly fix formatting and remove unnecessary elements Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:55:42 +02:00
Artem Navoiev	6370ae2d88	vmanomaly - models a bit pretify docs (#5618 ) * vmanomaly - models a bit pretify docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * typi Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix formatting Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:54:33 +02:00
Artem Navoiev	a90c9bf8c9	vmanomaly guides - fix formating, add missing piece, clarify statement, use common languagefor VM ecosystem (#5620 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:53:59 +02:00
Artem Navoiev	44ff8b3647	clarify cluster multitenacy in vmanomaly docs (#5619 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:53:30 +02:00
Artem Navoiev	f032f4347e	docs: fix markdown in how-to-monitor-k8s Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 21:52:59 +02:00
Aliaksandr Valialkin	0196902b2e	lib/promscrape/discovery/hetzner: fix golangci-lint warnings after `03a97dc678` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550	2024-01-16 21:51:48 +02:00
Roman Khavronenko	c69d5041bb	docs: mention requirements for `latest` backups (#5614 ) Add docs to explain that `latest` backup folder can be accessed more frequently than others. Hence, it is recommended to keep it on storages with low access price. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 21:28:27 +02:00
Aliaksandr Valialkin	9b449dfadf	app/vmstorage: deregister storage metrics before stopping the storage This prevents from possible nil pointer dereference issues when the storage metrics are read after the storage is stopped. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548	2024-01-16 21:26:03 +02:00
Aliaksandr Valialkin	9e5e514faf	lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() Previously the was a race condition when the background goroutine still could try collecting metrics from already stopped resources after returning from pushmetrics.Stop(). Now the pushmetrics.Stop() waits until the background goroutine is stopped before returning. This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549 and the commit `fe2d9f6646` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548	2024-01-16 21:18:22 +02:00
Aliaksandr Valialkin	fce76b2fb0	vendor/github.com/VictoriaMetrics/easyproto: update from v0.1.3 to v0.1.4 This fixes vet error for 32-bit architectures: https://github.com/VictoriaMetrics/VictoriaMetrics/actions/runs/7521709384/job/20472882877	2024-01-16 21:14:03 +02:00
Aleksandr Stepanov	3a6e3adc7d	vmagent: added hetzner sd config (#5550 ) * added hetzner robot and hetzner cloud sd configs * remove gettoken fun and update docs * Updated CHANGELOG and vmagent docs * Updated CHANGELOG and vmagent docs --------- Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-01-16 21:13:20 +02:00
Roman Khavronenko	d562d772a8	lib/storage: properly check for `storage/prefetchedMetricIDs` cache expiration deadline (#5607 ) Before, this cache was limited only by size. Cache invalidation by time happens with jitter to prevent thundering herd problem. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 21:08:59 +02:00
rbizos	62db64e71b	Handling negative index in Graphite groupByNode/aliasByNode (#5581 ) Handeling the error case with -1 Signed-off-by: Raphael Bizos <r.bizos@criteo.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2024-01-16 20:55:27 +02:00
Aliaksandr Valialkin	d566aa7d78	lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto	2024-01-16 20:48:30 +02:00
Aliaksandr Valialkin	063ea8c773	app/vmalert/remotewrite: properly calculate vmalert_remotewrite_dropped_rows_total It was calculating the number of dropped time series instead of the number of dropped samples. While at it, drop vmalert_remotewrite_dropped_bytes_total metric, since it was inconsistently calculated - at one place it was calculating raw protobuf-encoded sample sizes, while at another place it was calculating the size of snappy-compressed prompbmarshal.WriteRequest protobuf message. Additionally, this metric has zero practical sense, so just drop it in order to reduce the level of confusion.	2024-01-16 20:47:13 +02:00
Aliaksandr Valialkin	f7b589e38a	lib/prompb: switch to github.com/VictoriaMetrics/easyproto	2024-01-16 20:43:09 +02:00
Aliaksandr Valialkin	7d40506744	lib/prompb: change type of Label.Name and Label.Value from []byte to string This makes it more consistent with lib/prompbmarshal.Label	2024-01-16 20:41:37 +02:00
Aliaksandr Valialkin	8cb138e8df	lib/protoparser/datadogv2: simplify code for parsing protobuf messages after `0597718435` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451	2024-01-16 20:35:17 +02:00
Aliaksandr Valialkin	f8ae2abd88	lib/protoparser/opentelemetry: use github.com/VictoriaMetrics/easyproto for protobuf message unmarshaling and marshaling This reduces VictoriaMetrics binary size by 100KB. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2424	2024-01-16 20:34:18 +02:00
Aliaksandr Valialkin	9eef72bce9	lib/protoparser/datadogv2: add support for reading protobuf-encoded requests at /api/v2/series endpoint Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094	2024-01-16 20:32:15 +02:00
Dmytro Kozlov	b95d6f5f5e	app/vmctl: add insecure skip verify flags for source and destination addresses for native protocol (#5606 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5595	2024-01-16 20:29:41 +02:00
Zakhar Bessarab	178d64cc51	docs: explicitly mention "delete_series" endpoint accepts any HTTP method (#5605 ) See: #5552 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2024-01-16 20:14:04 +02:00
hagen1778	af47f96177	docs: make docs-sync Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 20:12:47 +02:00
Artem Navoiev	e1005209ba	docs: mention staleNaN handling during deduplication See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5587 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 20:11:45 +02:00
Dan Dascalescu	59ab3efc2b	docs: fix English in keyConcepts.md, add instant query use case (#5547 )	2024-01-16 19:54:09 +02:00
Fred Navruzov	2b372de326	docs: vmanomaly chapter hotfixes (#5583 ) * update link to book a demo for AD & RCA * fix invalid refs in components/model * - fix staging -> prod links - replace capitalized FAQ headers - change the section order on main page - replace :latest tag with current stable	2024-01-16 19:53:25 +02:00
Fred Navruzov	99c545bff8	- fix 404 in /guides page (#5582 ) - change back AD section title	2024-01-16 19:50:01 +02:00
Artem Navoiev	821ac6e374	docs: properly close diff Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 19:48:36 +02:00
Denys Holius	74c99549d6	docs/anomaly-detection/guides/README.md: fixed markdown link to the vmanomaly guide (#5578 )	2024-01-16 19:12:50 +02:00
Artem Navoiev	54723fc94e	docs: vmanomaly add list of guides to guides page Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 19:12:04 +02:00
Artem Navoiev	77570d83ee	docs: change sorting of anomaly deteciton Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 19:10:56 +02:00
Fred Navruzov	2b404742e2	docs: vmanomaly part 1 (#5558 ) * add `AD` section, fix links, release docs and changelog * - connect sections, refactor structure * - resolve suggestions - add FAQ section - fix dead links * - fix incorrect render of tables for Writer - comment out internal readers/writers - fix page ordering to some extent * - link licensing requirements from v1.5.0 to main page --------- Co-authored-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 19:09:37 +02:00
Denys Holius	c9b7452a86	CHANGELOG.md: fixed wrong links to vmalert-tool documentation page (#5570 )	2024-01-16 19:07:35 +02:00
Artem Navoiev	9152f91f49	docs: specify right link to grafana operator docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2024-01-16 17:45:21 +02:00
hagen1778	bcc3c9c12e	docs: fix typo in VictoriaLogs upgrading procedure Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 17:43:59 +02:00
hagen1778	583bbed48e	docs: docs-sync after `52692d001a` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 17:42:48 +02:00
hagen1778	14b9f7d71f	docs: add link to sandbox to the Grafana section Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-16 17:36:11 +02:00
Aliaksandr Valialkin	0114ade5eb	docs/keyConcepts.md: clarify which values can be stored in VictoriaMetrics without precision loss This is a follow-up for `43d7de4afe` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5485 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5503	2024-01-16 15:50:28 +02:00
hagen1778	f301dc5cfb	lib/uint64: remove accidentally added test Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-09 13:32:22 +01:00
hagen1778	2a7207f38a	app/all: follow-up after `84d710beab` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2024-01-09 13:17:09 +01:00
zhdd99	84d710beab	lib/pushmetrics: fix a panic caused by pushing metrics during the graceful shutdown process of vmstorage nodes. (#5549 ) Co-authored-by: zhangdongdong <zhangdongdong@kuaishou.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2024-01-09 13:01:03 +01:00
Artem Navoiev	5f10055f67	docs: vmagent specify default value for the compression level and its maximum. The question appers too often in our channel (#5575 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `095d982976`)	2024-01-08 20:15:05 +01:00
Dmytro Kozlov	37a76de800	app/vmui: fix broken link for the statistic inaccuracy explanation (#5568 ) (cherry picked from commit `105c6b2eb7`)	2024-01-08 20:15:04 +01:00
Dan Dascalescu	068610ac61	Link to start/end timestamp formats in url-examples (#5531 ) (cherry picked from commit `33df9bee22`)	2024-01-08 20:15:04 +01:00
hagen1778	12acea2584	dashboards: update cluster dashboard * add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components. * update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `463455665b`)	2024-01-08 11:58:56 +01:00
hagen1778	388894881e	vendor: go mod tidy & go mod vendor Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `910a39ad72`)	2023-12-22 16:10:33 +01:00
Hui Wang	c14e229b20	vmalert: automatically add `exported_` prefix for original evaluation… (#5398 ) automatically add `exported_` prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5161 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1f477aba41`)	2023-12-22 16:10:33 +01:00
Daria Karavaieva	372553479a	docs: vmanomaly guide v1.7.0 changes (#5505 ) (cherry picked from commit `be20501376`)	2023-12-22 16:10:32 +01:00
Dmytro Kozlov	2c0cbafc0c	docs: clarify information about values (#5503 ) (cherry picked from commit `43d7de4afe`)	2023-12-22 16:10:32 +01:00
hagen1778	f1ea68dca2	vendor/metrics: fix unaligned 64-bit atomic operation panic on 32-bit arch Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `34b69dcf58`)	2023-12-22 16:10:32 +01:00
hagen1778	b04bb9b140	docs: fix `unauthorizedAccessConfig` filed names for operator See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5520 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8ba483eca3`)	2023-12-22 16:10:31 +01:00
Aliaksandr Valialkin	12de0d39eb	lib/protoparser/datadogv2: take into account source_type_name field, since it contains useful value such as kubernetes, docker, system, etc.	2023-12-21 23:05:52 +02:00
Aliaksandr Valialkin	6feef14095	lib/protoparser: add missing /datadog/ prefix to the /api/v2/series path in the description for -datadog.maxInsertRequestSize command-line flag	2023-12-21 21:05:24 +02:00
Aliaksandr Valialkin	d145c68c5c	docs/CHANGELOG.md: typo fix after `fb90a56de2`: supperted -> supported	2023-12-21 21:01:54 +02:00
Aliaksandr Valialkin	62a105d9e9	app/{vminsert,vmagent}: preliminary support for /api/v2/series ingestion from new versions of DataDog Agent This commit adds only JSON support - https://docs.datadoghq.com/api/latest/metrics/#submit-metrics , while recent versions of DataDog Agent send data to /api/v2/series in undocumented Protobuf format. The support for this format will be added later. Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451	2023-12-21 20:50:27 +02:00
Roman Khavronenko	4837616df6	app/vmselect: drop `rollupDefault` function as duplicate (#5502 ) * app/vmselect: drop `rollupDefault` function as duplicate It is unclear why there are two identical fns `rollupDefault` and `rollupDistinct`. Dropping one of them. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update app/vmselect/promql/rollup.go * Update app/vmselect/promql/rollup.go --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-12-21 11:23:20 +02:00
Aliaksandr Valialkin	426a451435	lib/promauth: add more context to errors returned by Options.NewConfig() in order to simplify troubleshooting	2023-12-20 21:58:19 +02:00
Aliaksandr Valialkin	3a9cf13aaa	app/{vmagent,vmalert}: add the ability to set OAuth2 endpoint params via the corresponding *.oauth2.endpointParams command-line flags This is a follow-up for `5ebd5a0d7b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427	2023-12-20 21:38:16 +02:00
Morgan	64e96fccd9	Expose OAuth2 Endpoint Parameters to cli (#5427 ) The user may which to control the endpoint parameters for instance to set the audience when requesting an access token. Exposing the parameters as a map allows for additional use cases without requiring modification.	2023-12-20 21:38:13 +02:00
Aliaksandr Valialkin	c5f6ca268c	docs/Articles.md: add two articles about VictoriaMetrics from zetablogs.medium.com	2023-12-20 21:36:15 +02:00
Aliaksandr Valialkin	c888d76c4b	app/vmselect/netstorage: make sure that at least a single result is collected from every storage group before deciding whether it is OK to skip results from the remaining storage nodes	2023-12-20 19:53:49 +02:00
Nikolay	46a335aa1d	lib/awsapi: properly assume role with webIdentity token (#5495 ) * lib/awsapi: properly assume role with webIdentity token introduce new irsaRoleArn param for config. It's only needed for authorization with webIdentity token. First credentials obtained with irsa role and the next sts assume call for an actual roleArn made with those credentials. Common use case for it - cross AWS accounts authorization https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-12-20 19:07:04 +02:00
Aliaksandr Valialkin	261c173f4b	all: use Gauge instead of Counter for `*_config_last_reload_successful` metrics This allows exposing the correct TYPE metadata for these labels when the app runs with -metrics.exposeMetadata command-line flag. See https://github.com/VictoriaMetrics/metrics/pull/61#issuecomment-1860085508 for more details. This is follow-up for `326a77c697`	2023-12-20 14:25:44 +02:00
Yury Molodov	d0b047a2bf	vmui: add vmanomaly explorer (#5401 )	2023-12-20 14:15:25 +02:00
Dmytro Kozlov	7afec3d813	docs: remove default value from the `maxConcurrentInserts` flag (#5494 )	2023-12-20 14:15:20 +02:00
Aliaksandr Valialkin	0a99c819bf	all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page This may be needed for systems, which require this metadata such as Google Cloud Managed Prometheus. See https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type	2023-12-19 03:26:02 +02:00
Artem Navoiev	738a71b865	docs: vmalert exapand HA abbreviation Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-12-19 03:24:53 +02:00
hagen1778	49bd8adbc3	docs: mention `Splitting data streams` in `Multi-retention` guide Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-19 03:24:12 +02:00
hagen1778	0bb65bc83e	docs: add example for `Splitting data streams among multiple systems` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-19 03:23:24 +02:00
Aliaksandr Valialkin	9540d29154	lib/pushmetrics: add -pushmetrics.header and -pushmetrics.disableCompression command-line flags	2023-12-17 19:58:14 +02:00
Aliaksandr Valialkin	cbd8a62ba4	docs/CHANGELOG.md: typo fix: use proper backtick closing quote instead of single quote	2023-12-17 19:28:22 +02:00
Roman Khavronenko	bcb2b8247c	vmctl: rename `vm-native-disable-retries` to `vm-native-disable-per-metric-migration` (#5476 ) The change supposed to better reflect the meaning of this flag. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-17 19:19:22 +02:00
Aliaksandr Valialkin	76b120e355	lib/protoparser/opentelemetry: allow ingesting metrics without resource labels Some clients may ingest samples via OpenTelemetry protocol without Resource labels. Previously VictoriaMetrics was silently dropping such samples. The commit `317834f876` added vm_protoparser_rows_dropped_total{type="opentelemetry",reason="resource_not_set"} counter for tracking of such dropped samples. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5459 It is better from usability PoV to accept such samples instead of dropping them and incrementing the corresponding counter.	2023-12-17 19:16:43 +02:00
Aliaksandr Valialkin	bed19a0b20	docs/CHANGELOG.md: move the description of the bugfix from `9253c24dd6` into correct place The description of the bugfix was incorrectly placed in already released v1.96.0, while it should be placed in tip. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5450	2023-12-17 19:16:42 +02:00
Roman Khavronenko	5f14fa94dd	vmctl: retry requests that failed in the very end for `vm-native` (#5475 ) Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `664fa5cb78`)	2023-12-15 11:54:08 +01:00
Zakhar Bessarab	61f400eccb	lib/protoparser/opentelemetry: add metric to track skipped rows without resource (#5459 ) Currently, it is impossible to understand why metrics are not ingested when resource is not set by OTEL exporter. Adding metric should simplify debugging and make it improve debuggability. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `317834f876`)	2023-12-15 11:54:07 +01:00
Hui Wang	ed4f77575f	vmalert: validate schema for `-external.url` (#5450 ) Requests with wrong or no schema in `-external.url` could be rejected by alertmanager. So we validate schema on start up. (cherry picked from commit `9253c24dd6`)	2023-12-15 11:54:07 +01:00
Dima Lazerka	44c113f829	VMUI: Handle unknown query error response type (#5451 ) * VMUI: Handle unknown query error response type * vmui: add error text for unknown error type * Simplify nested `if`s for unknown error Accepting @Loori-R's suggestion Co-authored-by: Yury Molodov <yurymolodov@gmail.com> --------- Co-authored-by: Yury Moladau <yurymolodov@gmail.com> (cherry picked from commit `cd277e3f84`)	2023-12-15 11:54:07 +01:00
Artem Navoiev	c86e0fdc24	add link to RELEX solutions case study to home and signle pages Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-12-14 20:54:55 +02:00
nemobis	ac48e4dc9f	Add RELEX Solutions to case studies (#5469 ) Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-12-14 20:53:15 +02:00
Aliaksandr Valialkin	42629dead1	app/vmstorage: addd missing -inmemoryDataFlushInterval command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2023-12-14 20:47:49 +02:00
Aliaksandr Valialkin	329bd244d2	lib/fs: remove unused IsEmptyDir() This function became unused after the commit `43b24164ef` The unused function has been found with deadode tool - https://go.dev/blog/deadcode	2023-12-14 19:40:52 +02:00
Aliaksandr Valialkin	8bc87acfb9	docs/CHANGELOG.md: document the bugfix at `66c76a4d4d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414	2023-12-14 12:49:03 +02:00
Anton Tykhyy	51af1dfff7	Fix sum(aggr_over_time) 'got 1 args' error (#3028 ) (#5414 ) app/vmselect/promql/eval.go:evalAggrFunc shunts evaluation of AggrFuncExpr over rollupFunc over MetricsExpr to an optimized path. tryGetArgRollupFuncWithMetricExpr() checks whether expression can be shunted, but it mangles the AggrFuncExpr when the aggregation function has more than one argument. This results in queries like `sum(aggr_over_time("avg_over_time",m))` failing with error message 'expecting at least 2 args to "aggr_over_time"; got 1 args' while the analogous query `sum(avg_over_time(m))` executes successfully. This fix removes the unnecessary mangling. Signed-off-by: Anton Tykhyy <atykhyy@gmail.com>	2023-12-14 12:49:01 +02:00
Aliaksandr Valialkin	4ee42e9e73	app/vmauth: allow specifying an empty retry_status_codes and and zero drop_src_path_prefix_parts in order to override user-level setting Previously `retry_status_codes: []` and `drop_src_path_prefix_parts: 0` at `url_map` were equivalent to missing values. This was resulting in using the user-level values instead.	2023-12-14 01:06:50 +02:00
Aliaksandr Valialkin	51acf0179c	app/vmauth: add ability to route requests to different backends depending on the request host	2023-12-14 00:47:00 +02:00
Aliaksandr Valialkin	a57f125c67	docs/Single-server-VictoriaMetrics.md: cross-link HA docs with the corresponding chapter at vmauth docs	2023-12-13 23:07:09 +02:00
Aliaksandr Valialkin	b2dcb94322	deployment: update VictoriaMetrics images from v1.95.1 to v1.96.0 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0	2023-12-13 01:21:12 +02:00
Aliaksandr Valialkin	d59ee93d04	docs/CHANGELOG.md: cut v1.96.0 release	2023-12-13 00:43:04 +02:00
Aliaksandr Valialkin	37d64e24f9	vendor: run `make vendor-update`	2023-12-13 00:35:37 +02:00
Yury Molodov	e76c44c5b4	vmui: autocomplete usability improvements (#5422 ) * vmui: add show quick tip for autocomplete * vmui: auto-completion usability improvements #5348 * vmui: add const for min symbols in autocomplete * Use proper queries to VictoriaMetrics * vmui: fix comments for autocomplete * app/vmselect: run `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-12-13 00:33:27 +02:00
Aliaksandr Valialkin	e4bb2808f1	app/vmselect: add support for vmstorage groups with independent -replicationFactor per group Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197 See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmstorage-groups-at-vmselect Thanks to @zekker6 for the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/718	2023-12-13 00:14:34 +02:00
hagen1778	3b841fe9ce	app/vmctl: follow-up after `6af732b6f7` Make docs more clear about new feature. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `242472086b`)	2023-12-12 13:45:35 +01:00
Dmytro Kozlov	63fc200f16	app/vmctl: enable range steps in reverse order (#5444 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5376 (cherry picked from commit `6af732b6f7`)	2023-12-12 13:45:35 +01:00
hagen1778	14117f2f90	lib/promscrape: comsetic changes after `e373bb84d5` * fix typos in docs * add `shard-` prefix to generated links when `-promscrape.cluster.memberURLTemplate` is enabled Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e0fc5ef140`)	2023-12-12 13:45:34 +01:00
hagen1778	c2ca6cb84d	docs: fix formatting after a list in CHANGELOG.md Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1e02efd511`)	2023-12-12 13:45:34 +01:00
hagen1778	307bcb8d4d	app/vmctl: follow-up after `27668c9d01` * remove duplications in error messages * mention the change in CHANGELOG.md `27668c9d01` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `39c405ed4d`)	2023-12-11 15:38:22 +01:00
wozz	61f940591f	vmctl: check error in response from influxdb (#5446 ) (cherry picked from commit `27668c9d01`)	2023-12-11 15:38:20 +01:00
hagen1778	73f843a70e	docs: mention alerts change in CHANGELOG.md Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e13dc04fbf`)	2023-12-11 15:38:18 +01:00
hagen1778	383dce7201	alerts: simplify aggregation of alerting rules This is follow-up after `75196d7234` It updates some of the alerting rules to remove unnecessary aggregations. It keeps aggregations for expressions which are using multiple time series filters to make sure their label will match. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8fb68152e6`)	2023-12-11 15:38:16 +01:00
7840vz	231f6c7fb2	alerts: inverse grouping in vmagent alerts (#5429 ) Aggregations with by() have one sideeffect, that any custom labels you add to hosts are dropped too which can be used for alerts routing. Therefore, some good practice could be to use without() instead, with labels, like without(path) , or without(url) to get same aggregations but with any external labels left intact. (cherry picked from commit `75196d7234`)	2023-12-11 15:38:14 +01:00
Aliaksandr Valialkin	15a7542ef8	vendor: run `make vendor-update`	2023-12-11 10:48:47 +02:00
Aliaksandr Valialkin	050e15d85b	docs/CHANGELOG.md: document v1.93.9 LTS release	2023-12-11 10:39:38 +02:00
Aliaksandr Valialkin	09fe1255ca	docs/CHANGELOG.md: document v1.87.12	2023-12-10 14:30:30 +02:00
Aliaksandr Valialkin	842aba3f46	deployment/docker: update base Docker image from alpine:3.18.5 to alpine:3.19.0 See https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html	2023-12-10 02:28:31 +02:00
Aliaksandr Valialkin	de1b394af8	docs: sync -help output for VictoriaMetrics components after recent changes	2023-12-10 01:13:51 +02:00
Aliaksandr Valialkin	3d6517b05e	app/vmselect: add -search.maxResponseSeries command-line flag for limiting the number of time series a single response can return This limit can be used for preventing from high memory usage at Grafana when the response returns too many series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5372	2023-12-10 00:54:32 +02:00
Aliaksandr Valialkin	55eb48f5ee	app: make more clear that -tls enables https at -httpListenAddr	2023-12-10 00:25:23 +02:00
Aliaksandr Valialkin	6203c1a745	docs: follow-up after `49552eaa15` Link to the related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792 Fix heading for `Modifying HTTP headers` chapter at docs/vmagent.md	2023-12-08 23:56:40 +02:00
Aliaksandr Valialkin	49552eaa15	app/vmauth: add support for `hot standby` mode via `first_available` load balancing policy vmauth in `hot standby` mode sends requests to the first url_prefix while it is available. If the first url_prefix becomes unavailable, then vmauth falls back to the next url_prefix. This allows building highly available setup as described at https://docs.victoriametrics.com/vmauth.html#high-availability Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792	2023-12-08 23:32:10 +02:00
Aliaksandr Valialkin	475ae2a1be	lib/promscrape: add a wraning when the /service-discovery page contains incomplete list of dropped targets	2023-12-08 19:04:29 +02:00
dependabot[bot]	b68e116197	build(deps): bump actions/setup-go from 4 to 5 (#5435 ) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-12-08 18:15:59 +02:00
noodles2hg	f3c237bae1	lib/streamaggr/streamaggr.go: fix link in error message (#5439 )	2023-12-08 18:14:29 +02:00
Roman Khavronenko	276e9301f4	app/vmalert: sanitize label names before sending to Alertmanager (#5442 ) Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-08 18:09:07 +02:00
hagen1778	21c5bc5a98	docs: mentioned that re-routing increases number of active time series Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-08 18:08:08 +02:00
Aliaksandr Valialkin	2f02c87b02	docs: run `make docs-images-to-webp` after `02a0a7f428` This reduces added image sizes by 3x Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5437 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5370	2023-12-07 16:42:16 +02:00
Alexander Marshalov	e9cf39f519	added field `version` to the response for `/api/v1/status/buildinfo` API for using more efficient API in Grafana for receiving label values, added additional info about setup Grafana datasource (#5370 ) (#5437 )	2023-12-07 16:41:56 +02:00
Aliaksandr Valialkin	9074ab68d4	lib/promscrape: add `-promscrape.cluster.memberURLTemplate` command-line flag for creating direct links to vmagent instances at /service-discovery page See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018#issuecomment-1843811569	2023-12-07 16:05:03 +02:00
Aliaksandr Valialkin	32aea90847	app/vmselect/prometheus: go fmt after `b39e9257eb`	2023-12-07 16:05:01 +02:00
Aliaksandr Valialkin	9f79342e6a	app/vmselect/prometheus: properly encode Prometheus label values at /federate endpoint Prometheus spec says that only \, \n and " must be escaped inside label values. See `995743836e/content/docs/instrumenting/exposition_formats.md (L90)` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5431	2023-12-07 15:36:50 +02:00
hagen1778	77b9047a40	docs: fix wrong link in Troubleshooting.md Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `cb90d09c9d`)	2023-12-07 12:55:19 +01:00
Aliaksandr Valialkin	e500b5ef8c	docs/vmagent.md: mention that `vmagent` displays up to `-promscrape.maxDroppedTargets` at `/service-discovery` page Suggest increasing `-promscrape.maxDroppedTargets` command-line flag value if /service-discovery page misses some dropped targets. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018	2023-12-07 00:34:26 +02:00
Aliaksandr Valialkin	896a0f32cd	lib/promscrape: show -promscrape.cluster.memberNum values for vmagent instances, which scrape the given dropped target at /service-discovery page The /service-discovery page contains the list of all the discovered targets after the commit `487f6380d0` on all the vmagent instances in cluster mode ( https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets ). This commit improves debuggability of targets in cluster mode by providing a list of -promscrape.cluster.memberNum values per each target at /service-discovery page, which has been dropped becasue of sharding, e.g. if this target is scraped by other vmagent instances in the cluster. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4018	2023-12-07 00:11:30 +02:00
Aliaksandr Valialkin	12e94f10cc	deployment/docker: update Go builder from Go1.21.4 to Go1.21.5 See https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved	2023-12-06 22:33:27 +02:00
Aliaksandr Valialkin	e8dfecb3f1	lib/promscrape: show `never scraped` message for never scraped targets at /targets page	2023-12-06 22:33:27 +02:00
Dmytro Kozlov	6a41e1ec0c	app/vmalert: replace error metrics for gauges with counter metrics (#5217 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5160 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `935bec447b`)	2023-12-06 19:41:34 +01:00
Aliaksandr Valialkin	8b6bce61e4	lib/promscrape: follow-up for `97373b7786` Substitute O(N^2) algorithm for exposing the `vm_promscrape_scrape_pool_targets` metric with O(N) algorithm, where N is the number of scrape jobs. The previous algorithm could slow down /metrics exposition significantly when -promscrape.config contains thousands of scrape jobs. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5311 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5335	2023-12-06 17:36:48 +02:00
Aliaksandr Valialkin	509339bf63	app/vmselect: properly adjust the lower bound for the time range where raw samples must be selected for default_rollup() function Previously the lower bound could be too small, which could result in missing values at the beginning of the graph for default_rollup() function. This function is automatically applied to all the series selectors if they aren't explicitly wrapped into a rollup function - see https://docs.victoriametrics.com/MetricsQL.html#implicit-query-conversions While at it, properly take into account `-search.minStalenessInterval` command-line flag when adjusting the lower bound for the selected time range. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5388	2023-12-06 14:46:18 +02:00
Hui Wang	065f5a7f9e	vmagent: add `vm_promscrape_scrape_pool_targets` for scrape jobs like… (#5335 ) * vmagent: export `vm_promscrape_scrape_pool_targets` metric to track the number of targets that each scrape_job discovers * add extra panel for new metric	2023-12-06 14:46:02 +02:00
Artem Navoiev	716dee5de7	fix anchor in docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-12-06 14:40:41 +02:00
Aliaksandr Valialkin	559e4db512	Revert "add datadog /api/v2/series and /api/beta/sketches support (#5094 )" This reverts commit `d6b4c8e4ef`. Reason for revert: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094#issuecomment-1839789080	2023-12-05 02:30:40 +02:00
Aliaksandr Valialkin	61db92cdc7	Revert "lib/protoparser/datadog: follow-up after 543f218fe96574b9b2189c8350bb09afa349e3bb" This reverts commit `73d18fbc7a`. Reason for revert: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5094#issuecomment-1839789080	2023-12-05 02:29:00 +02:00
Aliaksandr Valialkin	bf187b2dc9	app/vmagent: add `-enableMultitenantHandlers` command-line flag This flag allows converting tenant id to (vm_account_id, vm_project_id) labels. this flag deprecates `-remoteWrite.multitenantURL` command-line flag, because `-enableMultitenantHandlers` is easier to use and combine with multitenant url at vminsert - https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels See https://docs.victoriametrics.com/vmagent.html#multitenancy Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1505	2023-12-05 01:35:59 +02:00
Aliaksandr Valialkin	5388d7ba12	docs/vmagent.md: mention that it may be useful to disable on-disk data persistence when reading data from Kafka or Google PubSub Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110	2023-12-04 23:13:23 +02:00
Aliaksandr Valialkin	85fcefaa34	app/vmagent: code cleanup for Kafka and Google PubSub consumers / producers - Add links to relevant docs into descriptions for every -kafka.* and -gcp.pubsub.* command-line flags. - Wait until message processing goroutines are stopped before returning from gcppubsub.Stop(). - Prevent from multiple calls to Init() without Stop(). - Drop message if tenantID cannot be parsed properly. - Take into account tenantID for all the supported message formats. - Support gzip-compressed messages for graphite format. - Use exponential backoff sleep when the message cannot be pushed to remote storage systems because of disabled on-disk persistence - https://docs.victoriametrics.com/vmagent.html#disabling-on-disk-persistence - Unblock from sleep as soon as Stop() is called. Previously the sleep could take up to 2 seconds after Stop() is called. - Remove unused globalCtx and initContext from app/vmagent/remotewrite/gcppubsub - Mention Google PubSub support at docs/enterprise.md - Make Google PubSub docs more clear at docs/vmagent.md This is a follow-up for commits 115245924a5f096c5a3383d6cc8e8b6fbd421984 and e6eab781ce42285a6a1750dc01eba6801dd35516 . Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/717 Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/713	2023-12-04 22:51:04 +02:00
Dmytro Kozlov	6770bad207	app/vmalert: expose `/vmalert/api/v1/rule` and `/api/v1/rule` API which returns rule status in JSON format (#5397 ) * app/vmalert: expose `/vmalert/api/v1/rule` and `/api/v1/rule` API which returns rule status in JSON format * app/vmalert: hide updates if query param not set * app/vmalert: fix panic (recursion call) * app/vmalert: add needed group name and file name * app/vmalert: fix comment, update behavior * app/vmalert: fix description * app/vmalert: simplify API for /api/v1/rule Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmalert: simplify API for /api/v1/rule Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmalert: simplify API for /api/v1/rule Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmalert: simplify API for /api/v1/rule Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmalert: simplify API for /api/v1/rule Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-12-04 22:49:39 +02:00
xzchaoo	758922e656	docs: fix the typo in vmctl.md (#5419 ) fix the typo in vmctl.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-12-04 18:17:53 +02:00
Aliaksandr Valialkin	a3d0bbfcda	deployment/docker: update backe Docker image from alpine 3.18.4 to 3.18.5 See https://www.alpinelinux.org/posts/Alpine-3.15.11-3.16.8-3.17.6-3.18.5-released.html	2023-12-04 18:17:07 +02:00
hagen1778	fd8731ed90	docs: fix formatting for datadog example Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-04 18:16:32 +02:00
Aliaksandr Valialkin	d868155751	app/vmselect: do not limit concurrency for static and fast queries Previously concurrency for static and fast queries was limited with the -search.maxConcurrentRequests command-line flag. This could complicate identifying heavy queries via `vmui` at `Top queries` and `Active queries` pages, since `vmui` and these pages couldn't be opened on overloaded vmselect. Thanks to @f41gh7 for the idea.	2023-12-04 18:14:29 +02:00
Aliaksandr Valialkin	b6d6a3a530	lib/promscrape: show dropped targets because of sharding at /service-discovery page Previously the /service-discovery page didn't show targets dropped because of sharding ( https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets ). Show also the reason why every target is dropped at /service-discovery page. This should improve debuging why particular targets are dropped. While at it, do not remove dropped targets from the list at /service-discovery page until the total number of targets exceeds the limit passed to -promscrape.maxDroppedTargets . Previously the list was cleaned up every 10 minutes from the entries, which weren't updated for the last minute. This could complicate debugging of dropped targets. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5389	2023-12-04 17:42:46 +02:00
Aliaksandr Valialkin	2f4dc2aff1	lib/backup: consistently use path.Join() when constructing paths for s3, gs and azblob E.g. replace `fs.Dir + filePath` with `path.Join(fs.Dir, filePath)` The fs.Dir is guaranteed to end with slash - see Init() functions. The filePath may start with slash. If it starts with slash, then `fs.Dir + filePath` constructs an incorrect path with double slashes. path.Join() properly substitutes duplicate slashes with a single slash in this case. While at it, also substitute incorrect usage of filepath.Join() with path.Join() for constructing paths to object storage systems, which expect forward slashes in paths. filepath.Join() substittues forward slashes with backslashes on Windows, so this may break creating or managing backups from Windows. This is a follow-up for 0399367be602b577baf6a872ca81bf0f99ba401b Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/719	2023-12-04 17:25:41 +02:00
Zakhar Bessarab	2992682f6c	lib/backup/s3remote: remove prev object versions for recursive delete (#719 ) * lib/backup/s3remote: remove prev object versions for recursive delete - fix error caused by sending empty objects list to be deleted. This was possible in case old versions of objects where deleted, but root-level entries where still available. This caused paginator to return an empty page which wasn't skipped. - delete previous versions of objects recursively for S3 remote Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/changelog: add vmbackupmanager fix entry Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/backup/s3remote: unify path construction for S3 objects Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-12-04 17:01:09 +02:00
Aliaksandr Valialkin	9f352f1b93	app/vminsert/newrelic: simplify the code a bit after `1fb8dc0092` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5416 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5421	2023-12-04 16:26:52 +02:00
Dmytro Kozlov	1fb8dc0092	app/vminsert: fix newrelic ingestion in cluster version (#5421 ) Properly pass tenant ID to ingested data from newrelic. Before tenant ID was mistakenly skipped.	2023-12-04 09:38:32 +01:00
hagen1778	c27968e79c	docs: follow-up after `760a530305` `760a530305` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e1359c904c`)	2023-12-01 14:02:00 +01:00
Artem Navoiev	fbd04e3437	docs: vmagent info about p queue disk size (#5399 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `760a530305`)	2023-12-01 14:01:40 +01:00
Hui Wang	3507e1e27b	vmalert-tool: fix alert_rule_test case when eval_time is not multiple of evaluation_interval (#5387 ) Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1911320c86`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-01 14:00:58 +01:00
Aliaksandr Valialkin	d1445bc0c8	all: expose additional metrics for simplifying debugging of VictoriaMetrics components Updates https://github.com/VictoriaMetrics/metrics/issues/54 (cherry picked from commit `8eddccfbb4`)	2023-12-01 14:00:28 +01:00
Aliaksandr Valialkin	e017176f45	docs/vmauth.md: add typical use cases (cherry picked from commit `837f6f0975`)	2023-12-01 14:00:23 +01:00
Aliaksandr Valialkin	f0215afee3	lib/promrelabel: add `keep_if_contains` and `drop_if_contains` relabeling actions (cherry picked from commit `ac65c6b178`)	2023-12-01 14:00:20 +01:00
Nikolay	9505d48070	lib/streamaggr: properly reference slice with labels (#5406 ) * lib/streamaggr: properly reference slice with labels by limiting slice capacity. It must fix issues with slice modification, in case of append new slice will be allocated, instead of modifying refrenced slice https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5402 * Reduce memory allocations when output_relabel_configs adds new labels to output samples --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> (cherry picked from commit `41f7940f97`)	2023-12-01 14:00:18 +01:00
Github Actions	2a0717c8db	Automatic update operator docs from VictoriaMetrics/operator@f628bee (#5407 ) (cherry picked from commit `7ca783dee9`)	2023-12-01 14:00:11 +01:00
Andrii Chubatiuk	4ed7da8c58	docs: sync mistakenly deleted docs from `543f218fe9` `543f218fe9` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `48228031e4`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-01 13:59:51 +01:00
hagen1778	73d18fbc7a	lib/protoparser/datadog: follow-up after `543f218fe9` * prevent /api/v1 from panic on parsing rows * add tests for Extract function for v1 and v2 api's * separate request types in different pools to prevent different objects mixing * add changelog line `543f218fe9` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `98d0f81f21`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-01 13:56:23 +01:00
Andrii Chubatiuk	d6b4c8e4ef	add datadog /api/v2/series and /api/beta/sketches support (#5094 ) Co-authored-by: Andrew Chubatiuk <andrew.chubatiuk@motional.com> Co-authored-by: Nikolay <https://github.com/f41gh7> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `543f218fe9`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-12-01 13:55:32 +01:00
hagen1778	1c5120ce67	docs: add 3rd party article "Observe and record performance of Spark jobs with Victoria Metrics" Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ec5b72c879`)	2023-11-29 16:43:19 +01:00
Ivan Yatskevich	bfaca07774	docs/dns-srv-typo-fix: replace `dns+src` with `dns+srv` (#5396 )	2023-11-28 17:46:00 +02:00
hagen1778	1e557b73a5	docs: mention contributor of PR 5368 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5424632ba3`)	2023-11-28 12:49:49 +01:00
luckyxiaoqiang	8ce82c5400	app/vmselect/promql: add day_of_year() function (#5368 ) Co-authored-by: dingxiaoqiang <dingxiaoqiang@bytedance.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `d7897e0d70`)	2023-11-28 12:49:48 +01:00
hagen1778	1ca672f3ac	docs: mention loadbalancer in Monitoring chapter Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8a0bb4bf17`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-28 12:49:48 +01:00
hagen1778	e443c20e92	docs: fix indentation for `/api/v1/labels` The indentation didn't change in `b51d16e74c` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d024fcf37f`)	2023-11-28 09:45:18 +01:00
hagen1778	46f5aeb7ab	docs: clarify steps for rollup cache purge for vmselects Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `5e38dde18d`)	2023-11-28 09:45:18 +01:00
hagen1778	562a2ddffc	docs: fix link for cache reset on vmselects Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f42ec79958`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-28 09:45:18 +01:00
Github Actions	4038a30d64	Automatic update operator docs from VictoriaMetrics/operator@e737115 (#5394 )	2023-11-27 12:02:33 +02:00
Aliaksandr Valialkin	f2346a79b6	docs/Cluster-VictoriaMetrics.md: document that multitenancy via labels is applied to data ingested via non-http protocols Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3009	2023-11-27 11:18:15 +02:00
Aliaksandr Valialkin	5ccc22d66d	app/vmagent: properly increase vmagent_remotewrite_samples_dropped_total when scraped samples cannot be sent to the remote storage and -remoteWrite.dropSamplesOnOverload is set This is a follow-up for `5034aa0773` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110	2023-11-25 14:44:42 +02:00
Aliaksandr Valialkin	e36f29080d	docs/vmagent.md: remove duplicate chapter for Google PubSub integration The previous chapter has been added in `752f89f13f` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5358	2023-11-25 14:44:41 +02:00
Aliaksandr Valialkin	2f14394335	app/vmagent: follow-up for `090cb2c9de` - Add Try* prefix to functions, which return bool result in order to improve readability and reduce the probability of missing check for the result returned from these functions. - Call the adjustSampleValues() only once on input samples. Previously it was called on every attempt to flush data to peristent queue. - Properly restore the initial state of WriteRequest passed to tryPushWriteRequest() before returning from this function after unsuccessful push to persistent queue. Previously a part of WriteRequest samples may be lost in such case. - Add -remoteWrite.dropSamplesOnOverload command-line flag, which can be used for dropping incoming samples instead of returning 429 Too Many Requests error to the client when -remoteWrite.disableOnDiskQueue is set and the remote storage cannot keep up with the data ingestion rate. - Add vmagent_remotewrite_samples_dropped_total metric, which counts the number of dropped samples. - Add vmagent_remotewrite_push_failures_total metric, which counts the number of unsuccessful attempts to push data to persistent queue when -remoteWrite.disableOnDiskQueue is set. - Remove vmagent_remotewrite_aggregation_metrics_dropped_total and vm_promscrape_push_samples_dropped_total metrics, because they are replaced with vmagent_remotewrite_samples_dropped_total metric. - Update 'Disabling on-disk persistence' docs at docs/vmagent.md - Update stale comments in the code Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5088 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110	2023-11-25 12:13:39 +02:00
Nikolay	25ac2aac31	app/vmagent: allow to disabled on-disk persistence (#5088 ) * app/vmagent: allow to disabled on-disk queue Previously, it wasn't possible to build data processing pipeline with a chain of vmagents. In case when remoteWrite for the last vmagent in the chain wasn't accessible, it persisted data only when it has enough disk capacity. If disk queue is full, it started to silently drop ingested metrics. New flags allows to disable on-disk persistent and immediatly return an error if remoteWrite is not accessible anymore. It blocks any writes and notify client, that data ingestion isn't possible. Main use case for this feature - use external queue such as kafka for data persistence. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110 * adds test, updates readme * apply review suggestions * update docs for vmagent * makes linter happy --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-25 12:12:29 +02:00
Aliaksandr Valialkin	3d22f98344	vendor: update github.com/VictoriaMetrics/fastcache from v1.12.1 to v1.12.2 This should help reducing GC overhead growth at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5379	2023-11-24 13:39:32 +02:00
Aliaksandr Valialkin	3674232128	docs: make more visible that the maximum JSON line length, which is accepted by /api/v1/import, is limited by -import.maxLineLen command-line flag value This is a follow-up for `0cf55ded34` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5364	2023-11-24 13:14:40 +02:00
Roman Khavronenko	26242f526e	lib/protoparser: decrease `import.maxLineLen` from 100MB to 10MB (#5364 ) Tests showed that importing a single line with 70MB size takes 5.3GiB RSS memory for VictoriaMetrics single-node. In the scenario when user exports and imports data from one VM to another, it could possibly lead to OOM exception for destination VM. Importing a single line with 16MB size taks 1.3GiB RSS memory. Hence, the limit for `import.maxLineLen` was decreased from 100MB to 10MB to improve reliability of VictoriaMetrics during imports. Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-24 13:13:33 +02:00
Aliaksandr Valialkin	01bc62eff9	docs/CHANGELOG.md: document Google PubSub support at vmagent (see `752f89f13f` )	2023-11-23 21:14:04 +02:00
Nikolay	892889823a	apply review comments (#5358 )	2023-11-23 21:13:57 +02:00
Github Actions	3a96830ed6	Automatic update operator docs from VictoriaMetrics/operator@fec3f9d (#5381 )	2023-11-23 21:06:27 +02:00
Aliaksandr Valialkin	a906a7d85c	app/vmagent/remotewrite: do not drop persistent queues when -remoteWrite.multitenantURL is set It is unsafe to drop persistent queues when -remoteWrite.multitenantURL command-line flag is set, since these queues are created on demand when a new sample for the given tenant is pushed to the remote storage. This addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357 The issue has been appeared in the commit `f3a51e8b1d` when implementing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014	2023-11-23 20:43:21 +02:00
Github Actions	28df725a37	Automatic update operator docs from VictoriaMetrics/operator@45bfa36 (#5373 )	2023-11-22 20:25:46 +02:00
Aliaksandr Valialkin	10b4dfbbf9	app/vmalert/notifier: remove backticks from the description for -notifier.blackhole command-line flag Backticks in flag description are automatically converted to flag type. See https://pkg.go.dev/flag#PrintDefaults This is a follow-up for `20025d4fd6` and `25317b4e70`	2023-11-22 20:17:45 +02:00
Aliaksandr Valialkin	db6dadf1f7	docs: convert png images to webp in all the docs except of docs/operator/* This reduces the size of docs/* folder from 33MB to 18MB Images inside docs/operator/* must be converted at the https://github.com/VictoriaMetrics/operator/tree/master/docs and then the updated images must be automatically propagated to the docs/operator/* This is a follow-up for `d3f919df3e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5206	2023-11-22 19:29:47 +02:00
Github Actions	bf934482c5	Automatic update operator docs from VictoriaMetrics/operator@b96131b (#5371 )	2023-11-22 19:23:15 +02:00
Aliaksandr Valialkin	0ccc1aca0a	deployment/docker: remove built binaries at bin folder after creating docker image from them at `make publish-via-docker`	2023-11-21 14:33:50 +02:00
Aliaksandr Valialkin	fc40ebba7a	.github/workflows/codeql-analysis.yml: cache Go artifacts	2023-11-21 13:05:15 +02:00
Aliaksandr Valialkin	2ebd5cbb53	.github/workflows/main.yml: ignore changes inside dashboards and deployment/.yml The dashaboards/ and deployment/.yml do not contain files, which may change main workflow results, so it is better to ignore them.	2023-11-21 12:58:43 +02:00
hagen1778	ae6152be5f	lib/storage: fix typo Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-21 12:22:49 +02:00
Aliaksandr Valialkin	d8aceda5fd	.github/workflows: run build and test jobs in parallel in order to speed up the workflow run	2023-11-21 12:20:31 +02:00
hagen1778	91e365acb6	lib/storage: fix typo Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-21 12:10:34 +02:00
Aliaksandr Valialkin	4f876ada0e	.github/workflows: add Go version to Go artifacts cache key When Go version changes, artifacts for the previous Go version may becomes useless, so there is a little sense in re-using them.	2023-11-21 12:02:33 +02:00
Aliaksandr Valialkin	9acd591c8e	.github/workflows: take into account Makefile contents when generating cache key for Go build aftifacts The cache key must change when the corresponding 'make ...' command changes inside Makefile.	2023-11-21 11:51:56 +02:00
Aliaksandr Valialkin	b71388c1a8	.github/workflows: use `stable` Go release - it should always point to the latest stable release This eliminates the need to update .github/workflows/* files whenever new Go stable release is out, like in the `2db1a664e1` . See https://github.com/actions/setup-go#using-stableoldstable-aliases	2023-11-21 11:39:04 +02:00
Aliaksandr Valialkin	3cb33196e6	.github/workflows/main.yml: try improving caching for Go artifacts The default caching for Go artifacts from actions/setup-go@v4 uses the hash of go.sum file as a cache key - see https://github.com/actions/cache/blob/main/examples.md#go---modules . This isn't enough for VictoriaMetrics case, since different makefile actions build different the Go artifacts, which need to be cached. So embed the action name in the cache key.	2023-11-21 01:12:17 +02:00
Aliaksandr Valialkin	95b076ba99	.github/workflows/codeql-analysis.yml: remove `check-latest` and `cache` inputs for `actions/setup-go` - The `cache: true` is no longer needed starting from actions/setup-go@v4 - see https://github.com/actions/setup-go#v4 - The `check-latest: true` may slow down the action, so it is better to disalbe it - see https://github.com/actions/setup-go#check-latest-version	2023-11-21 01:12:00 +02:00
Aliaksandr Valialkin	c285fca256	Makefile: allow specifying the needed concurrency for make via MAKE_CONCURRENCY env var	2023-11-21 01:11:36 +02:00
Aliaksandr Valialkin	6b75523468	Makefile: speedup `release`, `publish` and `crossbuild` rules by using parallel make	2023-11-20 23:07:11 +02:00
Aliaksandr Valialkin	46e58f3669	app/vmagent/README.md: sync with docs/vmagent.md after `cbe4a5c251` , so `make docs-sync` properly works	2023-11-20 22:43:28 +02:00
Nikolay	c06044ef52	app/vmagent: adds google pubsub as remoteWrite dst and ingest consumer (#713 ) it allows to push and receive metrics from google pubsub queue Adds needed documentation and examples for it	2023-11-20 22:43:26 +02:00
hagen1778	34b7783461	docs: calrify version when vminsertConnsShutdownDuration was added Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-20 17:14:57 +01:00
hagen1778	0dbbffbdd5	docs: typo after `3f5a41e35e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `20025d4fd6`)	2023-11-20 17:06:21 +01:00
hagen1778	6575d646c0	docs: follow-up after `d3f919df3e` `d3f919df3e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3ffa8975d4`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-20 11:54:28 +01:00
Dmytro Kozlov	2362af3b0c	docs/managed-victoriametrics: use webp format to reduce image size (#5206 ) (cherry picked from commit `d3f919df3e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-20 11:53:06 +01:00
Khanh Quoc Le	03e5ebaea9	Add _stream fields log (#5068 )	2023-11-17 16:04:13 +01:00
Hui Wang	91379331eb	lib/protoparser/promremotewrite: fall back to zstd decoding if Snappy-decoding fails (#5344 ) This case is possible after the following steps: 1. vmagent successfully performed handshake with the -remoteWrite.url and the remote storage supports zstd-compressed data. 2. remote storage became unavailable or slow to ingest data, vmagent compressed the collected data into blocks with zstd and puts these blocks to persistent queue on disk. 3. vmagent restarts and the remote storage is unavailable during the handshake, then vmagent falls back to Snappy compression. 4. vmagent starts sending zstd-compressed data from persistent queue to the remote storage, while falsely advertizing it sends Snappy-compressed data. 5. The remote storage receives zstd-compressed data and fails unpacking it with Snappy. The solution is the same as `12cd32fd75`, just fall back to zstd decompression if Snappy decompression fails.	2023-11-17 15:53:18 +01:00
Aliaksandr Valialkin	39c56e8f65	docs/enterprise.md: update VictoriaMetrics version in examples from v1.95.0 to v1.95.1 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1	2023-11-17 15:47:24 +01:00
Github Actions	2e6c404cdd	Automatic update operator docs from VictoriaMetrics/operator@388745c (#5340 )	2023-11-17 15:47:22 +01:00
Aliaksandr Valialkin	1149a98873	deployment: update VictoriaMetrics docker image tag from v1.95.0 to v1.95.1 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1	2023-11-17 15:43:43 +01:00
Aliaksandr Valialkin	5492ccf0d5	app/vmselect/promql: reduce the number of memory allocations inside copyTimeseriesShallow() Previously the number of memory allocations inside copyTimeseriesShallow() was equal to 1+len(tss) Reduce this number to 2 by pre-allocating a slice of timeseries structs with len(tss) length.	2023-11-17 15:41:38 +01:00
luckyxiaoqiang	3419c14b35	docs/metricsql: remove duplicate sentence (#5349 )	2023-11-17 15:41:21 +01:00
Artem Navoiev	acfaf5c352	gh action bump pagefind version to 1.0.4 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-17 15:40:54 +01:00
Aliaksandr Valialkin	1a15b0f57b	docs/CHANGELOG.md: cut v1.95.1	2023-11-16 20:32:27 +01:00
Aliaksandr Valialkin	8723c8546a	vendor: run `make vendor-update`	2023-11-16 20:21:16 +01:00
Aliaksandr Valialkin	5c43f2261e	dashboards: remove `path!="/favicon.ico"` filter from `requests rate` graphs The `path!="/favicon.ico"` filter has little sense, since there are many other special paths, which may be filtered out - /metrics, /flags, /health, /ping, /robots.txt, /-/healthy, /-/ready, /reload, etc. See /lib/httpserver/httpserver.go for more details. It will be hard or impossible to maintain filters for all these paths, so it is better to drop this filter in order to simplify queries and improve the consistency of these queries.	2023-11-16 19:29:46 +01:00
Aliaksandr Valialkin	994b3da361	app/vmselect: simplify code a bit after `63e0f16062` Use only a single call to prometheus.WriteErrorResponse() inside sendPrometheusError	2023-11-16 18:15:08 +01:00
Aliaksandr Valialkin	5ad1c2b46a	docs/FAQ.md: add a link to https://docs.victoriametrics.com/#monitoring in questions where this is needed	2023-11-16 17:45:38 +01:00
Aliaksandr Valialkin	633ec37022	app/vmselect/promql: typo fix after `7ca8ebef20` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332	2023-11-16 17:01:19 +01:00
Aliaksandr Valialkin	a0f02d06d7	lib/handshake: typo fix after `ef80a89a24`: SetReadDeadline -> SetWriteDeadline Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5327	2023-11-16 16:47:07 +01:00
Aliaksandr Valialkin	ef80a89a24	lib/handshake: add SetReadDeadline and SetWriteDeadline implementations additionally to SetDeadline This is a follow-up for `27a5461785` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5327	2023-11-16 16:43:36 +01:00
Roman Khavronenko	27a5461785	lib/handshake: check for deadline in Read and Write methods (#5327 ) The buffered connection could have exceeded the underlying connection deadline during reading or writing to an internal buffer. With this change, buffered connection struct additionally checks for a deadline in Read/Write methods. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-16 16:33:40 +01:00
Github Actions	f430d506f2	Automatic update operator docs from VictoriaMetrics/operator@bc8b02f (#5331 )	2023-11-16 16:29:56 +01:00
Roman Khavronenko	c0039ce7a3	docs/vmalert: clarify deduplication recommendations for HA setup (#5336 ) Please see discussion here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5279 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-16 16:27:47 +01:00
Aliaksandr Valialkin	147fe45828	docs/CHANGELOG.md: remove duplicate word `query` after `2cbdb1db22`	2023-11-16 16:24:15 +01:00
Aliaksandr Valialkin	7ca8ebef20	app/vmselect/promql: properly handle duplicate series when merging cached results with the results obtained from the database evalRollupFuncNoCache() may return time series with identical labels (aka duplicate series) when performing queries satisfying all the following conditions: - It must select time series with multiple metric names. For example, {__name__=~"foo\|bar"} - The series selector must be wrapped into rollup function, which drops metric names. For example, rate({__name__=~"foo\|bar"}) - The rollup function must be wrapped into aggregate function, which has no streaming optimization. For example, quantile(0.9, rate({__name__=~"foo\|bar"}) In this case VictoriaMetrics shouldn't return `cannot merge series: duplicate series found` error. Instead, it should fall back to query execution with disabled cache. Also properly store the merged results. Previously they were incorrectly stored because of a typo introduced in the commit `41a0fdaf39` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5337	2023-11-16 16:16:17 +01:00
Github Actions	16a41593e6	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@95d0711 (#5329 )	2023-11-16 15:44:27 +01:00
hagen1778	7d72474a38	dashboards: use `version` instead of `short_version` in annotations `version` label won't show the difference if various flavors of the same version were deployed. But `short_version` will. For example, on the sandbox env we test VM builds before new version release. Without this change, the version update won't be visible on dashboard. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d389a4fcf3`)	2023-11-16 09:27:42 +01:00
Aliaksandr Valialkin	844b0db67a	docs/Articles.md: add https://hackernoon.com/unleashing-vm-histograms-for-ruby-migrating-from-prometheus-to-victoriametrics-with-vm-client	2023-11-16 00:53:53 +01:00
Yury Molodov	cc5f1745ca	vmui: change autocomplete hotkey to Alt/Option + A (#5328 )	2023-11-15 23:33:33 +01:00
Aliaksandr Valialkin	68c0038a5d	docs/vmbackup.md: fix links to https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages This is a follow-up for `2fc7e9f47e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121	2023-11-15 23:27:00 +01:00
Aliaksandr Valialkin	9a1354e8a9	docs/vmagent.md: refer to proper command-line flag: -remoteWrite.shardByURL.labels instead of -remoteWrite.shardByURLLabels This is a follow-up for `ed70a40669` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942	2023-11-15 23:03:30 +01:00
Aliaksandr Valialkin	f9355d34be	docs: mention that VictoriaMetrics and vmagent support data ingestion via New Relic protocol now This is a follow-up for `f60c08a7bd` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520	2023-11-15 22:56:34 +01:00
Aliaksandr Valialkin	772d2081ca	docs/Release-Guide.md: point to the proper location for CHANGELOG.md at github.com/VictoriaMetrics/operator repository	2023-11-15 21:56:11 +01:00
Aliaksandr Valialkin	656a6862d7	deployment: update reference to VictoriaLogs from v0.4.1-victorialogs to v0.4.2-victorialogs	2023-11-15 20:47:14 +01:00
Aliaksandr Valialkin	bc3a52cf50	docs/VictoriaLogs/README.md: cut v0.4.2-victorialogs	2023-11-15 20:47:14 +01:00
Aliaksandr Valialkin	bafe62b487	deployment: update references to VictoriaMetrics components from v1.94.0 to v1.95.0	2023-11-15 20:38:51 +01:00
Aliaksandr Valialkin	8b01c6caf4	app/vmalert-tool: add missing multiarch directory This is needed for 'make publish-vmalert-tool'	2023-11-15 18:13:05 +01:00
Aliaksandr Valialkin	9ad4a8fffe	docs/CHANGELOG.md: cut v1.95.0 release	2023-11-15 17:46:02 +01:00
Aliaksandr Valialkin	6cbfe316bb	vendor: update github.com/klasuspost/compress from v1.17.2 to v1.17.3 See https://github.com/klauspost/compress/releases/tag/v1.17.3	2023-11-15 17:19:01 +01:00
Aliaksandr Valialkin	bd5bbdf00c	docs/CHANGELOG.md: document v1.93.8 LTS release	2023-11-15 17:12:56 +01:00
Aliaksandr Valialkin	60ff3cbb3d	lib/querytracer: add missing blank comment line after `3121d76bee`	2023-11-15 16:11:50 +01:00
Aliaksandr Valialkin	de3d5943eb	docs/stream-aggregation.md: clarify that stream aggregation is applied after all the configured relabeling This is a follow-up after `68d2cb203d`	2023-11-15 15:54:57 +01:00
Aliaksandr Valialkin	6a8911ad38	docs/CHANGELOG.md: document v1.87.11 LTS release	2023-11-15 15:54:57 +01:00
hagen1778	14df5af660	docs/stream-aggr: specify the relabeling order during aggregation Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `68d2cb203d`)	2023-11-15 14:29:29 +01:00
Aliaksandr Valialkin	9d3f1ec0d0	app/vmctl/README.md: sync with docs/vmctl.md after `7b2e2a23c2`	2023-11-15 12:58:31 +01:00
Aliaksandr Valialkin	00f8c55efd	Makefile: remove package-base dependency from `publish` rule, since this dep is set inside all the `publish-*` dependencies This is a follow-up for `d4099a75be`	2023-11-15 12:58:30 +01:00
John Belmonte	e94ec36ef6	vmctl README.md typo (#5326 )	2023-11-15 12:57:49 +01:00
John Belmonte	7cd9757a2d	relabeling.md: fix link (#5325 )	2023-11-15 12:57:49 +01:00
Aliaksandr Valialkin	c245161ca2	vendor: run `make vendor-update`	2023-11-14 22:45:17 +01:00
Aliaksandr Valialkin	e9639a49c2	lib/ingestserver: properly log the number of closed connections Previously there was off-by-one error, which resulted in logging len(conns-1) connections instead of len(conns) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922	2023-11-14 21:53:10 +01:00
Aliaksandr Valialkin	d7a63529b5	docs/CHANGELOG.md: consistently prepend command-line flags with a single dash	2023-11-14 21:44:46 +01:00
hagen1778	cfc58dd932	docs: clarify vmalert flag changes Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-14 21:44:46 +01:00
Aliaksandr Valialkin	d8220066dc	docs/Cluster-VictoriaMetrics.md: clarify how -storage.vminsertConnsShutdownDuration command-line flag works	2023-11-14 21:41:19 +01:00
Nikolay	0730c2586d	lib/querytracer: makes package concurrent safe to use (#5322 ) * lib/querytracer: makes package concurrent safe to use it must fix various issues with concurrent code usage. Especially, when it's not reasonable to wait for all goroutines to be finished * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-14 20:58:28 +01:00
Aliaksandr Valialkin	5b7f40907e	app/vmselect/netstorage: do not retry request when deadline is exceeded	2023-11-14 19:57:29 +01:00
Aliaksandr Valialkin	1f7ab894d7	lib/logger: increase default -loggerMaxArgLen command-line flag value from 500 to 1000 The 500 chars limit for the maximum arg lengths during logging appeared to be too low for some cases	2023-11-14 19:55:55 +01:00
Artem Navoiev	83275af54b	docs: update Grafana setup section, use more direct link and add noti… (#5287 ) (cherry picked from commit `5d61a7327d`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-14 14:36:35 +01:00
hagen1778	72a40539b0	dashboards: update description for RSS and anonymous memory panels to be consistent for single-node, cluster and vmagent dashboards. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d3ae2b2f62`)	2023-11-14 10:00:11 +01:00
hagen1778	777424082b	deployment/dashboards: respect `job` and `instance` filters for `alerts` annotation in cluster and single-node dashboards Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d6ae082598`)	2023-11-14 10:00:11 +01:00
Aliaksandr Valialkin	2f885d8e57	app/vmselect/promql: typo fixes after `7cf7740d18`	2023-11-14 03:34:25 +01:00
Aliaksandr Valialkin	3a487666ca	lib/ingestserver: typo fix after `f7834767c1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922	2023-11-14 03:26:04 +01:00
Aliaksandr Valialkin	9ff1ee333f	app/vmselect/promql: properly handle instant query optimization conrner cases for min_over_time() and max_over_time() - If min_over_time(m[offset] @ timestamp) <= min_over_time(m[offset] @ (timestamp-window)), then the optimization can be applied. - If max_over_time(m[offset] @ timestamp) >= max_over_time(m[offset] @ (timestamp-window)), then the optimization can be applied.	2023-11-14 02:58:18 +01:00
Yury Molodov	0fe02e8d9d	vmui: reduced the number of server requests (#5253 ) * vmui: reduced the number of server requests * run `make vmui-update vmui-logs-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-14 01:50:57 +01:00
Aliaksandr Valialkin	d6a2264709	docs/CHANGELOG.md: document `0e056ddb2d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5203	2023-11-14 01:24:29 +01:00
Yury Molodov	33e65e2cab	vmui: fix trailing slash in serverURL (#5271 ) * vmui: add function to autoremove slash at the end of serverURL (#5203) * vmui: change removeTrailingSlash func	2023-11-14 01:24:29 +01:00
Noah Labrecque	fbb572a180	fix: apply correct bounds to sf and tf (#5274 )	2023-11-14 01:19:47 +01:00
Aliaksandr Valialkin	9760221214	lib/logstorage: always check the previous indexBlockHeader for blocks with matching tenantID and/or streamID The previous indexBlockHeader may contain blocks for the matching tenantID and/or streamID, so it must be scanned unconditionally during the search. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856 This is a follow-up for `89dcbc2fe7`	2023-11-14 01:02:02 +01:00
XLONG96	77033dbfb6	lib/logstorage: fix streamID and tenantID search (#4856 ) (#5295 )	2023-11-14 01:02:02 +01:00
Zakhar Bessarab	f7834767c1	vmcluster: re-routing enhancement (#5293 ) * app/vmstorage: close vminsert connections gradually before stopping storage Implements graceful shutdown approach suggested here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1768146878 Test results for this can be found here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1790640274 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmstorage: update graceful shutdown logic - close connections from vminsert in determenistic order - update flag description - lower default timeout to 25 seconds. 25 seconds value was chosen because the lowest default value used in default configuration deployments is 30s(default value in Kubernetes and ansible-playbooks). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/cluster: add information about re-routing enhancement during restart Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/changelog: add entry for new command-line flag Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * {app/vmstorage,lib/ingestserver}: address review feedback Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/cluster: add note to update workload scheduler timeout Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * wip --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-14 01:00:42 +01:00
Aliaksandr Valialkin	c1f651a9f9	app/vmauth: add ability to drop the specified number of `/`-delimited prefix parts from request path This can be done via `drop_src_path_prefix_parts` option at `url_map` and `user` levels. See https://docs.victoriametrics.com/vmauth.html#dropping-request-path-prefix	2023-11-13 22:34:40 +01:00
Aliaksandr Valialkin	12cd32fd75	lib/protoparser/promremotewrite: fall back to Snappy decoding if zstd decoding fails This case is possible after the following steps: 1. vmagent tries to perform handshake with the -remoteWrite.url in order to determine whether the remote storage supports zstd-compressed data. 2. The remote storage is unavailable during the handshake. In this case vmagent falls back to Snappy compression for the data sent to the remote storage. 3. vmagent compresses the collected data into blocks with Snappy and puts these blocks to persistent queue on disk. 4. The remote storage becomes available. 5. vmagent restarts, performs the handshake with the remote storage and detects that it supports zstd-compressed data. 6. vmagent starts sending Snappy-compressed data from persistent queue to the remote storage, while falsely advertizing it sends zstd-compressed data. 7. The remote storage receives Snappy-compressed data and fails unpacking it with zstd. The solution is to just fall back to Snappy decompression if zstd decompression fails. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301	2023-11-13 21:25:39 +01:00
Aliaksandr Valialkin	356deada8c	lib/htmlcomponents: use relative links for the top page and for favicon.ico This allows hiding VictoriaMetrics components behind proxies with arbitrary path prefixes. For example, vmagent HTTP handlers can be served via /vmagent/ path prefix: - http://proxy/vmagent/targets - http://proxy/vmagent/service-discovery The path prefix can be arbitrary. For example, below are vmagent urls for /tenantID/vmagent/ path prefix: - http://proxy/tenantID/vmagent/targets - http://proxy/tenantID/vmagent/service-discovery While at it, consistently serve favicon.ico from any path directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5306 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5307	2023-11-13 20:28:17 +01:00
Aliaksandr Valialkin	a45cbc101f	all: cleanup: remove `// +build ...` lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:15:42 +01:00
Aliaksandr Valialkin	32bee8b40f	vendor: run `make vendor-update`	2023-11-13 18:54:33 +01:00
Aliaksandr Valialkin	fb2071a01e	lib/regexutil: properly handle alternate regexps surrounded by .+ or .* Previously the following regexps were improperly handled: .+foo\|bar.+ .foo\|bar. This could lead to unexpected regexp match results. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297 Thanks to @Haleygo for the initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5308	2023-11-13 18:25:57 +01:00
Aliaksandr Valialkin	8256937d5e	docs/VictoriaLogs/CHANGELOG.md: follow-up for `66527c5981` Document the change Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5312 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300	2023-11-13 10:40:19 +01:00
Yury Molodov	695bc7ff36	vmui: ui logs enhancements (#5312 ) * vmui/logs: fix time sorting #5300 * vmui/logs: add base query validation * vmui/logs: add a message for empty results	2023-11-13 10:40:18 +01:00
Aliaksandr Valialkin	22927dcc53	lib/stringsutil: add tests for LimitStringLen() function	2023-11-13 10:33:07 +01:00
Dmytro Kozlov	faf788b4a6	lib/stringsutil: fix failing test (#5313 ) We have failed test on master branch. ``` --- FAIL: TestFormatLogMessage (0.00s) logger_test.go:24: unexpected result; got "foo: abcde, \"foo bar baz\", xx" want "foo: a..e, \"f..z\", xx" ``` if failed because maxArgs maxLen <= 4 in the `LimitStringLen` in that case we always will return the income string but in the test we limit the maxLen by value 4 ``` f("foo: %s, %q, %s", []interface{}{"abcde", fmt.Errorf("foo bar baz"), "xx"}, 4, `foo: a..e, "f..z", xx`)	2023-11-13 10:33:06 +01:00
Aliaksandr Valialkin	2a3352c70e	docs/CHANGELOG.md: remove trailing whitespace after `bffd30b57a`	2023-11-13 09:47:36 +01:00
Aliaksandr Valialkin	54c494ae8e	docs/vmauth.md: add missing dashes in front of command-line flags at the `Backend TLS setup` section Dashes must be consistently used in front of command-line flags across the documentation. This is a follow up for `61594d2bd8`	2023-11-13 09:45:52 +01:00
Aliaksandr Valialkin	b9aba7edfb	app/vmauth: properly pass `Host` header to backends Previously the `Host` header was remained unchanged when passing it in requests to backends. This may improperly work if the backend uses host-based routing. While at it, allows http/2.0 requests to backends. While VictoriaMetrics components do not accept http/2.0 requests, other backends can require such requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240	2023-11-13 09:45:34 +01:00
Aliaksandr Valialkin	78bc816220	app/vmauth: follow-up for `323f3720ed` - Re-use identically configured http.Transport across multiple users. This fixes handling of the limit on the number of connection, which can be established per each backend via -maxIdleConnsPerBackend command-line flag. This limit stopped working after `323f3720ed` - Add docs about backend TLS setup at https://docs.victoriametrics.com/vmauth.html#backend-tls-setup - Add ability to disable backend TLS verification for all the users via -backend.tlsInsecureSkipVerify command-line flag. This flag may be useful when -auth.config contains big number of users, and every user must disable backend TLS verification. - Add ability to specify TLS Root CA via tls_ca_file option at per-user basis and via -backend.tlsCAFile command-line flag across all the users. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240	2023-11-13 09:45:16 +01:00
Aliaksandr Valialkin	4e621aaa0b	docs/Articles.md: typo fix	2023-11-13 09:44:53 +01:00
Aliaksandr Valialkin	76384b6d28	app/vmauth: improve docs a bit after `323f3720ed` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240	2023-11-13 09:44:25 +01:00
Aliaksandr Valialkin	bf12a49087	app/vmagent/README.md: sync with docs/vmagent.md after `930d26b2ff`	2023-11-13 09:44:07 +01:00
Aliaksandr Valialkin	d9ecc3f6d7	lib/logger: add `-loggerMaxArgLen` command-line flag for fine-tuning the maximum length of logged args	2023-11-13 09:43:49 +01:00
Aliaksandr Valialkin	c916294b61	app/vmselect/promql: optimize instant queries with min_over_time() and max_over_time() rollup functions This is a follow-up for `41a0fdaf39`	2023-11-13 09:43:18 +01:00
Aliaksandr Valialkin	7bbdecb79a	deployment: update Go builder from Go1.21.3 to Go1.21.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved	2023-11-13 09:40:08 +01:00
Aliaksandr Valialkin	ed79f9806a	lib/blockcache: do not cache entries, which were attempted to be accessed 1 or 2 times Previously entries which were accessed only 1 time weren't cached. It has been appeared that some rarely executed heavy queries may read indexdb block twice in a row instead of once. There is no need in caching such a block then. This change should eliminate cache size spikes for indexdb/dataBlocks when such heavy queries are executed. Expose -blockcache.missesBeforeCaching command-line flag, which can be used for fine-tuning the number of cache misses needed before storing the block in the caching.	2023-11-13 09:38:57 +01:00
Aliaksandr Valialkin	f859b33808	docs/Articles.md: sort third-party articles by importance	2023-11-13 09:36:59 +01:00
Aliaksandr Valialkin	bc0b339507	docs/Articles.md: add a link to https://blog.cloudflare.com/introducing-http-traffic-anomalies-notifications/	2023-11-13 09:36:41 +01:00
Aliaksandr Valialkin	18c5f20e50	docs/Single-server-VictoriaMetrics.md: make `High availability` section more clear	2023-11-13 09:36:16 +01:00
PhracturedBlue	6675d08ebc	Support building images via podman (#4978 )	2023-11-13 09:35:02 +01:00
Zakhar Bessarab	7c7e0a5caa	docs/changelog: document vmbackupmanager bugfix (#5303 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-11-13 09:31:58 +01:00
Artem Navoiev	10435a44be	docs: vmagent change the codeblock languages Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-13 09:31:21 +01:00
Github Actions	bd29f2f1e2	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@cc18249 (#5305 )	2023-11-13 09:30:45 +01:00
Github Actions	0cb932e11d	Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@52bdb4a (#5304 )	2023-11-13 09:30:21 +01:00
Roman Khavronenko	becf7bf8df	app/vmalert: update remote-write process (#5284 ) * app/vmalert: update remote-write process * automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers. * increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded. * increment `vmalert_remotewrite_dropped_rows_total` amd `vmalert_remotewrite_dropped_bytes_total` metrics if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update docs/CHANGELOG.md --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Hui Wang <haley@victoriametrics.com>	2023-11-13 09:25:29 +01:00
Artem Navoiev	63d4787f60	github actions: sync docs use the latest hugo version in CI Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-13 09:21:25 +01:00
Artem Navoiev	4bba3e39bd	docs: url example change the title h2->h3 h3->h4 for better indexing in search Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-13 09:20:55 +01:00
Artem Navoiev	b72dc10bb3	docs: fix formatting in stream aggregation more Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-13 09:20:23 +01:00
Artem Navoiev	55df212a76	docs: fix formatting in stream aggregation Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-13 09:19:49 +01:00
Github Actions	a01cc78f28	Automatic update operator docs from VictoriaMetrics/operator@b4b79da (#5291 )	2023-11-13 09:18:53 +01:00
hagen1778	10da9e6e01	app/vmalert: fix typo in `remoteWrite.concurrency` description Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c07dc45786`)	2023-11-03 22:05:00 +01:00
Yury Molodov	d7c6153f68	vmui: display query error on Explore metrics page (#5272 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5202 (cherry picked from commit `f90d2ec843`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-11-03 16:25:21 +01:00
hagen1778	cf3812bd0b	docs: make docs-sync after `323f3720ed` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `054367c421`)	2023-11-03 16:24:37 +01:00
Zakhar Bessarab	dea4695df5	app/vmauth: add option to skip TLS verification (#5256 ) Add `tls_insecure_skip_verify` option on per-user basis which allows to disable TLS verification for all requests to backend on behalf of this user. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `323f3720ed`)	2023-11-03 12:05:26 +01:00
Aliaksandr Valialkin	996e746c2c	Makefile: update golangci-lint version from v1.54.2 to v1.55.1 See https://github.com/golangci/golangci-lint/releases/tag/v1.55.1	2023-11-02 21:42:35 +01:00
Aliaksandr Valialkin	6bdb8cf2d0	vendor: run `make vendor-update`	2023-11-02 21:03:35 +01:00
Aliaksandr Valialkin	f202962c55	go.mod: pin the latest working version of golang.org/x/exp	2023-11-02 20:55:41 +01:00
Aliaksandr Valialkin	3d6f4da3b3	docs: update -help output after recent changes to VictoriaMetrics components	2023-11-02 20:27:16 +01:00
Aliaksandr Valialkin	bf01a97f17	docs/CHANGELOG.md: update the description of the optimization for SLO/SLI-like queries according to latest changes See commits `4497a08e3d` and `92826b0b4a`	2023-11-02 20:09:22 +01:00
Roman Khavronenko	4e8c762fd9	app/vmalert: add label `file` pointing to the group's filename to metrics (#5281 ) The filename should help identifying alerting rules belonging to specific groups with identical names but different filenames. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b5254199c6`)	2023-11-02 16:02:29 +01:00
hagen1778	3773510e8f	app/vmalert: verify alert name correctness in restore test Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6eb205f8b0`)	2023-11-02 16:02:29 +01:00
Hui Wang	44fcdf0cf0	vmalert: reduce restore query request for each alerting rule (#5265 ) reduce the number of queries for restoring alerts state on start-up. The change should speed up the restore process and reduce pressure on `remoteRead.url`. (cherry picked from commit `90d45574bf`)	2023-11-02 16:02:28 +01:00
Aliaksandr Valialkin	7fc5178a4b	app/vmselect/promql: add missing trace message in rollupResultCache.GetSeries()	2023-11-02 09:17:13 +01:00
Aliaksandr Valialkin	44227c0287	docs/CHANGELOG.md: typo fix: tis -> this	2023-11-02 08:33:48 +01:00
Aliaksandr Valialkin	95f5984aae	docs/Single-server-VictoriaMetrics.md: document why data inside `<-storageDataPath>/snapshots` directory should be manipulated only via snapshot API	2023-11-02 08:31:08 +01:00
Aliaksandr Valialkin	c04e667f9d	docs/CHANGELOG.md: document v1.93.7 LTS release	2023-11-02 08:21:10 +01:00
Aliaksandr Valialkin	369d37749d	app/vmagent/remotewrite: add -remoteWrite.shardByURL.labels command-line flag This command-line flag can be used for specifying a list of labels used for sharding among -remoteWrite.url entries when -remoteWrite.shardByURL command-line flag is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942	2023-11-01 23:09:08 +01:00
Alexander Marshalov	ffeec24811	vmauth: add browser authorization request for http requests without… (#5234 ) * vmauth: add browser authorization request for http requests without credentials to a route that is not in the `unauthorized_user` section (when `unauthorized_user` is specified). * add link to issue in CHANGELOG * Extend vmauth docs * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-11-01 21:00:52 +01:00
Aliaksandr Valialkin	ece7024f11	app/vmselect/promql: reduce the minimum lookbehind window for enabling SLO/SLI optimizations from 24 hours to 6 hours This reduction is based on production testing. Also expose -search.minWindowForInstantRollupOptimization command-line flag, so users could fine-tune this arg for their needs	2023-11-01 20:19:19 +01:00
Aliaksandr Valialkin	e4365dbe3e	app/vmselect: run `make quicktemplate-gen` after `b8739bc00b`	2023-11-01 17:53:30 +01:00
Github Actions	86aae00a60	Automatic update operator docs from VictoriaMetrics/operator@49826be (#5270 ) Co-authored-by: Alexander Marshalov <_@marshalov.org>	2023-11-01 17:50:50 +01:00
Artem Navoiev	23d09684a6	add Try new Docs button in the current docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-11-01 17:50:27 +01:00
Aliaksandr Valialkin	ae9b4c94bc	app/vmselect: return stats.seriesFetched as string instead of number vmalert expects string value for stats.seriesFetched, so it is impossible switching to number without breaking compatibility with old vmalert releases :( It is still unclear why stats.seriesFetched has string type in the first place...	2023-11-01 17:49:28 +01:00
Aliaksandr Valialkin	6a98f9df54	app/vmui: show query execution duration in the header of query input field This should simplify the process of query optimization	2023-11-01 16:46:42 +01:00
Hui Wang	4fafdda13e	vmalert: support specifying full http url in notifier static_configs target (#5261 ) * vmalert: support specifying full http or https urls in notifier static_configs target address * show right label results in ui	2023-11-01 16:44:54 +01:00
Aliaksandr Valialkin	c5e3b11762	app/vmselect/promql: apply SLO-like optimization to all the `count_*_over_time()` functions This is a follow-up for `41a0fdaf39`	2023-11-01 09:58:50 +01:00
Aliaksandr Valialkin	b96d55e1e4	app/vmselect/promql: typo fix, which could lead to panic during range query execution The panic is: BUG: unexpected values after merging new values This is a follow-up for `41a0fdaf39`	2023-11-01 09:58:50 +01:00
Github Actions	51afab3c7f	Automatic update operator docs from VictoriaMetrics/operator@57c1bf6 (#5266 )	2023-11-01 09:58:50 +01:00
Aliaksandr Valialkin	28f0610e14	app/vmui: fix non-working `Disable cache` checkbox at `JSON` and `Table` views	2023-10-31 22:58:15 +01:00
Aliaksandr Valialkin	7b7ad44e84	app/vmselect/promql: properly calculate rollup result if lookbehind window isn't set This is a follow-up for `41a0fdaf39`	2023-10-31 22:23:04 +01:00
Aliaksandr Valialkin	744f8c3fe7	app/vmselect/promql: add outliers_iqr(q) and outlier_iqr_over_time(m[d]) functions These functions allow detecting anomalies in series and samples using Interquartile range method. See Outliers section at https://en.wikipedia.org/wiki/Interquartile_range for more details.	2023-10-31 22:14:14 +01:00
Aliaksandr Valialkin	48b842d2ad	vendor: run `make vendor-update`	2023-10-31 20:20:07 +01:00
Aliaksandr Valialkin	9661918bb4	app/vmselect/promql: optimize repeated SLI-like instant queries with lookbehind windows >= 1d Repeated instant queries with long lookbehind windows, which contain one of the following rollup functions, are optimized via partial result caching: - sum_over_time() - count_over_time() - avg_over_time() - increase() - rate() The basic idea of optimization is to calculate rf(m[d] @ t) as rf(m[offset] @ t) + rf(m[d] @ (t-offset)) - rf(m[offset] @ (t-d)) where rf(m[d] @ (t-offset)) is cached query result, which was calculated previously The offset may be in the range of up to 1 hour.	2023-10-31 20:08:38 +01:00
Aliaksandr Valialkin	9ba007a636	app/vmselect/promql: wrap too long line after `a950873fff`	2023-10-31 19:11:05 +01:00
Aliaksandr Valialkin	5e7d495eb1	lib/httpserver: follow-up for `0638bbe69c` - Replace spaces with underscores in the `reason` label value for the vm_http_request_errors_total metric in order be consistent with Prometheus-like naming - Clarify the description for the change at docs/CHANGELOG.md Updates https://github.com/victoriaMetrics/victoriaMetrics/issues/4590 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5166	2023-10-31 19:10:48 +01:00
Aliaksandr Valialkin	2288f81c5b	lib/persistentqueue: properly re-create flock.lock file inside directory if persistent queue is broken. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5249 Thanks to @Sniper91 for the bugreport and initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5233	2023-10-31 19:10:26 +01:00
Aliaksandr Valialkin	09c5ac238a	lib/httpserver: call Request.Header() only once instead of calling it each time a new request header is set This is a follow-up for `ad839aa492`	2023-10-31 19:10:09 +01:00
Artem Navoiev	fc7e2e887e	github actions: fix typo in hugo version Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-31 19:09:48 +01:00
Artem Navoiev	0293a7033c	github actions: use 0.119 hugo version as far latest contains bug Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-31 19:09:19 +01:00
Aliaksandr Valialkin	a7a73b9845	docs/Cluster-VictoriaMetrics.md: clarify the description on why -dosnwampling.period must be set at both vmstorage and vmselect This is a follow-up for `ca7457d906`	2023-10-31 19:08:21 +01:00
Aliaksandr Valialkin	ffcd757533	docs/Single-server-VictoriaMetrics.md: cosmetic fixes after `23369321f1`	2023-10-31 19:05:45 +01:00
Aliaksandr Valialkin	40a53b516d	docs/CHANGELOG.md: move the description for -http.header.* command-line flags from SECURITY to FEATURE The SECURITY label should be applied only to changes, which fix security issues. The change at `ad839aa492` adds new command-line flags, which can be used for improving security in some cases. They do not fix any security issues. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5111	2023-10-31 19:05:01 +01:00
Aliaksandr Valialkin	c22b63af04	lib/storage: follow-up for `29cebd82fb` Use atomic.CompareAndSwapUint32() instead of atomic.LoadUint32() followed by atomic.StoreUint32(). This makes the code more clear. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159	2023-10-31 19:03:50 +01:00
hagen1778	8c3bac8f40	dashboards/cluster: fix description about `max` threshold for `Concurrent selects` panel. Before, it was mistakenly implying that `max` is equal to the double of available CPUs. Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5214 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-31 19:03:21 +01:00
Roman Khavronenko	9d8f93050c	app/vmselect: expose `vm_memory_intensive_queries_total` counter metric (#5208 ) The new metric gets increased each time `-search.logQueryMemoryUsage` memory limit is exceeded by a query. This metric should help to identify expensive and heavy queries without inspecting the logs. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-31 19:02:22 +01:00
hagen1778	f9c7822588	docs: follow-up for `0638bbe69c` `0638bbe69c` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a8051d48c4`)	2023-10-31 12:54:57 +01:00
venkatbvc	85fd4917b1	vmauth: add counter metrics for auth successes and failures (#5166 ) New labels `reason="wrong basic auth creds"` and `reason="wrong auth key"` were added to metric `vm_http_request_errors_total` to help identify auth errors. https://github.com/victoriaMetrics/victoriaMetrics/issues/4590 Co-authored-by: Rao, B V Chalapathi <b_v_chalapathi.rao@nokia.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `0638bbe69c`)	2023-10-31 12:54:57 +01:00
hagen1778	9debdb497c	dashboards/vmalert: add new panel `Missed evaluations` The new panel supposed to indicate alerting groups that miss their evaluations. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `aaf9e3d526`)	2023-10-31 10:35:57 +01:00
hagen1778	659171686c	deployment/alerts: add `TooManyMissedIterations` alerting rule The new rule for vmalert supposed to detect groups that miss their evaulations due to slow queries. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9866974a53`)	2023-10-31 10:35:57 +01:00
hagen1778	497c708aaa	dashboards: fix `Errors rate to Alertmanager` filter The panel `Errors rate to Alertmanager` had `group` label filter applied to the expression, while the metric `vmalert_alerts_send_errors_total` doesn't have that label. This resulted into always empty results. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8874b525b7`)	2023-10-31 10:35:57 +01:00
Roman Khavronenko	afd160f1dc	docs: mention information loss when downsampling gauges (#5204 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `23369321f1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-30 19:05:39 +01:00
Roman Khavronenko	c6d74108fb	docs: explain motivation behind having `-downsampling.period` on vmselect (#5205 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-30 15:29:35 +01:00
Hui Wang	8a786e5df4	vmalert: fix alert firing state in replay mode (#5192 ) fix possible missing firing states for alerting rules in replay mode Before if one firing stage is bigger than single query request range, like rule with a big `for`, alerting rule won't able to be detected as firing. Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `abcb21aa5e`)	2023-10-30 13:55:48 +01:00
hagen1778	f0d10e2004	docs/troubleshooting: mention issue with un-ordered labels See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5219#issuecomment-1773441711 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e964df8039`)	2023-10-30 13:55:45 +01:00
hagen1778	14b1997659	docs: rm mention of default values for security HTTP headers The headers, their corresponding flags are mentioned at https://docs.victoriametrics.com/#security Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a64b37cf24`)	2023-10-30 11:46:49 +01:00
Dima Lazerka	ed8fc04898	lib/httpserver: add flags to specify HSTS / Frame-Options / CSP headers for httpserver (#5111 ) support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options` HTTP headers in all VictoriaMetrics components. The values for headers can be specified by users via the following flags: `-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`. Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ad839aa492`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-30 11:41:38 +01:00
Roman Khavronenko	733b73ffed	lib/storage: log warning about RO mode only on state change (#5191 ) Before, vmstorage would log the same message each second producing excessive amount of logs. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `29cebd82fb`)	2023-10-30 11:29:49 +01:00
Aliaksandr Valialkin	a66c261b55	app/vmui: change the order of tables at `Top queries` tab Move the most interesting table - queries with the most summary time to execute - to the top	2023-10-28 11:57:08 +02:00
Aliaksandr Valialkin	15dda54e79	lib/promscrape/discovery/kubernetes: propagate possible errors at newAPIWatcher() to the caller This allows substituting FATAL panics with recoverable runtime errors such as missing or invalid TLS CA file and/or missing/invalid /var/run/secrets/kubernetes.io/serviceaccount/namespace file. Now these errors are logged instead of PANIC'ing, so they can be fixed by updating the corresponding files without the need to restart vmagent. This is a follow-up for `90427abc65` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5243	2023-10-27 20:27:58 +02:00
Hui Wang	a37125d043	lib/promscrape/discovery/kubernetes: avoid possible panic if given caFile under kubernetes.SDConfig.HTTPClientConfig is not exist (#5243 ) follow up `d5a599badc`	2023-10-27 20:27:58 +02:00
Aliaksandr Valialkin	20aeb8b65d	lib/promscrape/discovery/kubernetes: stop all the url watchers, which belong to a particular groupWatcher, at once Previously url watchers for pod, service and node objects could be mistakenly closed when service discovery was set up only for endpoints and endpointslice roles, since watchers for these roles may start start pod, service and node url watchers with nil apiWatcher passed to groupWatcher.startWatchersForRole(). Now all the url watchers, which belong to a particular groupWatcher, are stopped at once when this groupWatcher has no apiWatcher subscribers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216 The issue has been introduced in v1.93.5 when addressing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850	2023-10-27 14:34:25 +02:00
Hui Wang	69f4a58f76	do not print redundant error logs when failed to scrape consul or no… (#5239 ) * do not print redundant error logs when failed to scrape consul or nomad target prometheus performs the same because it uses consul lib which just drops the error(`1806bcb38c/api/api.go (L1134)`)	2023-10-27 14:18:47 +02:00
hagen1778	ddedeb1d42	app/vmalert: remove unclear comment The timestamp alignment should be applied as a last step to keep the timestamp consistent. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-27 14:09:01 +02:00
Daria Karavaieva	076a796061	model list - isolation forest (#5235 ) * model list - isolation forest * curse of dimensionality * isol forest definition change, minor fixes * blank line fix	2023-10-27 14:08:30 +02:00
Aliaksandr Valialkin	8fbe5a0893	lib/promscrape: do not add a suggestion for enabling TCP6 in error message when the dial address is TCPv4	2023-10-27 14:06:49 +02:00
Dima Lazerka	1e48ad486e	Revert "lib/promscrape: do not add a suggestion for enabling TCP6 in error message when the dial address is TCPv4" It broke CI (lint) This reverts commit `5464376d16`.	2023-10-27 14:06:31 +02:00
Aliaksandr Valialkin	46dd504d81	lib/promscrape: do not add a suggestion for enabling TCP6 in error message when the dial address is TCPv4	2023-10-26 09:56:55 +02:00
Aliaksandr Valialkin	af6dc9c963	lib/promscrape: properly track the number of updated service discovery routines inside Config.mustRestart() This is a follow-up for `d5a599badc`	2023-10-26 09:56:36 +02:00
Aliaksandr Valialkin	b3bcf99cea	lib/promauth: typo fix in the error message after `d5a599badc`: obtaine -> obtain	2023-10-26 09:56:20 +02:00
Aliaksandr Valialkin	f03e81c693	lib/promauth: follow-up for `e16d3f5639` - Make sure that invalid/missing TLS CA file or TLS client certificate files at vmagent startup don't prevent from processing the corresponding scrape targets after the file becomes correct, without the need to restart vmagent. Previously scrape targets with invalid TLS CA file or TLS client certificate files were permanently dropped after the first attempt to initialize them, and they didn't appear until the next vmagent reload or the next change in other places of the loaded scrape configs. - Make sure that TLS CA is properly re-loaded from file after it changes without the need to restart vmagent. Previously the old TLS CA was used until vmagent restart. - Properly handle errors during http request creation for the second attempt to send data to remote system at vmagent and vmalert. Previously failed request creation could result in nil pointer dereferencing, since the returned request is nil on error. - Add more context to the logged error during AWS sigv4 request signing before sending the data to -remoteWrite.url at vmagent. Previously it could miss details on the source of the request. - Do not create a new HTTP client per second when generating OAuth2 token needed to put in Authorization header of every http request issued by vmagent during service discovery or target scraping. Re-use the HTTP client instead until the corresponding scrape config changes. - Cache error at lib/promauth.Config.GetAuthHeader() in the same way as the auth header is cached, e.g. the error is cached for a second now. This should reduce load on CPU and OAuth2 server when auth header cannot be obtained because of temporary error. - Share tls.Config.GetClientCertificate function among multiple scrape targets with the same tls_config. Cache the loaded certificate and the error for one second. This should significantly reduce CPU load when scraping big number of targets with the same tls_config. - Allow loading TLS certificates from HTTP and HTTPs urls by specifying these urls at `tls_config->cert_file` and `tls_config->key_file`. - Improve test coverage at lib/promauth - Skip unreachable or invalid files specified at `scrape_config_files` during vmagent startup, since these files may become valid later. Previously vmagent was exitting in this case. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959	2023-10-26 09:55:47 +02:00
Aliaksandr Valialkin	8c9e3b7b50	lib/promscrape/discovery/kubernetes/kubeconfig_test.go: make TestParseKubeConfigSuccess test code easier to follow	2023-10-26 09:54:40 +02:00
Aliaksandr Valialkin	02684a0b29	lib/promauth: properly parse string contents for ca, cert and key fields at tls_config Previously yaml parser wasn't accepting string values for these fields, because it was mistakenly expecting a list of uint8 values instead.	2023-10-26 09:54:18 +02:00
Aliaksandr Valialkin	194deeea1b	lib/promscrape: move duplicate code from functions, which collect ScrapeWork lists for distinct SD types into Config.getScrapeWorkGeneric() This removes more than 200 lines of duplicate code	2023-10-26 09:53:59 +02:00
Aliaksandr Valialkin	19940b5629	app/vmalert/config: fix flacky test TestParseBad It could return either `failed to read` or `failed to parse` errors depending on whether the given url can be loaded or not under the current environment	2023-10-26 09:53:40 +02:00
Aliaksandr Valialkin	36a1fdca6c	all: consistently use %w instead of %s in when error is passed to fmt.Errorf() This allows consistently using errors.Is() for verifying whether the given error wraps some other known error.	2023-10-26 09:44:40 +02:00
Aliaksandr Valialkin	dc6a718a52	lib/workingsetcache: fix outdated comments for Load() and New() functions	2023-10-26 09:43:04 +02:00
Artem Navoiev	3b00e7b599	github-actions: build search index during the doc sync (#5224 ) * github-actions: build search index during the doc sync Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-26 09:40:46 +02:00
Alexander Marshalov	cf42a080af	lib/streamaggr: respect `streamAgg.dropInput` with empty stream aggr config (#5213 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207	2023-10-26 09:30:12 +02:00
krakazyabra	c85ff74ac3	docs/case-studies: update Wedos info (#5211 )	2023-10-26 09:26:31 +02:00
Aliaksandr Valialkin	94e061087f	docs: use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest instead of https://github.com/VictoriaMetrics/VictoriaMetrics/releases link where needed The https://github.com/VictoriaMetrics/VictoriaMetrics/releases link may show non-latest releases at the top, such as LTS releases or VictoriaLogs releases. So it is better to use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest link, which always redirect to the latest available release of VictoriaMetrics.	2023-10-26 09:23:17 +02:00
Aliaksandr Valialkin	190beb79d3	lib/storage: fix test TestStorageSeriesAreNotCreatedOnStaleMarkers	2023-10-26 09:22:15 +02:00
Roman Khavronenko	cd2247b24a	app/vmselect: limit the number of parallel workers by 32 (#5195 ) * app/vmselect: limit the number of parallel workers by 32 The change should improve performance and memory usage during query processing on machines with big number of CPU cores. The number of parallel workers for query processing is controlled via `-search.maxWorkersPerQuery` command-line flag. By default, the number of workers is limited by the number of available CPU cores, but not more than 32. The limit can be increased via `-search.maxWorkersPerQuery`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip - The `-search.maxWorkersPerQuery` command-line flag doesn't limit resource usage, so move it from the `resource usage limits` to `troubleshooting` chapter at docs/Single-server-VictoriaMetrics.md - Make more clear the description for the `-search.maxWorkersPerQuery` command-line flag - Add the description of `-search.maxWorkersPerQuery` to docs/Cluster-VictoriaMetrics.md - Limit the maximum value, which can be passed to `-search.maxWorkersPerQuery`, to GOMAXPROCS, because bigger values may worsen query performance and increase CPU usage - Improve the the description of the change at docs/CHANGELOG.md. Mark it as FEATURE instead of BUGFIX, since it is closer to a feature than to a bugfix. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-10-26 09:15:27 +02:00
Yury Molodov	45501eccab	vmui: update dependencies (#5194 )	2023-10-26 09:08:05 +02:00
Aliaksandr Valialkin	76606eed22	docs/Articles.md: add an article https://rtfm.co.ua/en/victoriametrics-vmauth-proxy-authentication-and-authorization/	2023-10-26 09:06:54 +02:00
Aliaksandr Valialkin	52d7464095	docs/FAQ.md: add questions on why VictoriaMetrics doesnt rebalance data and doesnt restore replication factor between vmstorage nodes	2023-10-26 09:06:25 +02:00
Aliaksandr Valialkin	d3cab41426	docs/FAQ.md: refresh the answer to the question about how does VictoriaMetrics compare to competing solutions - Mention Grafana Mimir - Fix broken links	2023-10-26 09:06:01 +02:00
Github Actions	a35ae0716e	Automatic update operator docs from VictoriaMetrics/operator@2c826bb (#5188 )	2023-10-26 09:05:33 +02:00
hagen1778	afab547821	lib/storage: follow-up after `188cfe3a85` `188cfe3a85` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-26 09:04:49 +02:00
Ilya Trefilov	1fd3385965	lib/storage: do not create tsid if metric contains stale marker(#5069 ) (#5174 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5069	2023-10-26 09:03:19 +02:00
Hui Wang	d7dd7614eb	fix inconsistent behaviors with prometheus when scraping (#5153 ) * fix inconsistent behaviors with prometheus when scraping 1. address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959. skip job with wrong syntax in `scrape_configs` with error logs instead of exiting; 2. show error messages on vmagent /targets ui if there are wrong auth configs in `scrape_configs`, previously will print error logs and do scrape without auth header; 3. don't send requests if there are wrong auth configs in: 1. vmagent remoteWrite; 2. vmalert datasource/remoteRead/remoteWrite/notifier. * add changelogs * address review comments * fix ut	2023-10-26 08:56:54 +02:00
hagen1778	f00729ee24	app/vmalert: fix typo in tests Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c07909a20b`)	2023-10-26 08:55:20 +02:00
hagen1778	cf541c757a	app/vmalert: fix tests after `a216fe6728` `a216fe6728` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `eed0c3c6b0`)	2023-10-26 08:55:06 +02:00
Hui Wang	855c25b6c4	remove vmalert-tool code from branch cluster (#5229 ) Follow up `130e0ea5f0`. vmalert-tool can't be easily adapted for vmcluster now, cause it needs to set up the whole vmcluster[vminsert+vmstorage+vmselect] first. You can use vmalert-tool to run unit tests for alerting and recording rules. It will perform the following actions: - sets up an isolated VictoriaMetrics instance; - simulates the periodic ingestion of time series; - queries the ingested data for recording and alerting rules evaluation like vmalert; But component packages have functions that not exported and variables with same name, so to implement this for cluster will need amount of code refactor and doesn't look like a good thing to themselves. So I want to remove it from the cluster branch.	2023-10-25 14:48:11 +02:00
hagen1778	6c63ca18f5	app/vmalert: follow-up after `c9375cac5e` `c9375cac5e` Descriptions were updated in attempt to make it more clear for readers, re-phrasing and linking missing docs. `eval_delay` was added to tests to verify it can be unmarshalled. `eval_delay` is now applied before timestamp alignment to make it more predictable. Before, if delay < interval the timestamp won't be aligned. `eval_delay` and `eval_offset` was added to API output. `PreviouslySentSeriesToRW` converted to private `previouslySentSeriesToRW`. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a216fe6728`)	2023-10-25 14:39:49 +02:00
Hui Wang	86d861ec55	vmalert: add `-rule.evalDelay` flag and `eval_delay` as group attribute (#5185 ) Also mark `-datasource.lookback` as will be deprecated, see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155. (cherry picked from commit `c9375cac5e`)	2023-10-25 14:39:49 +02:00
hagen1778	d9118cdaab	deployment/alerts: update `TooHighMemoryUsage` annotation The memory usage isn't measured on 5m interval anymore. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4e0a779efe`)	2023-10-25 14:39:48 +02:00
hagen1778	d349d6a9ce	deployment/alerts: make `TooHighMemoryUsage` more tolerable to spikes Using `min_over_time` should reduce the amount of false positives when component is running in near-the-threshold state. Now it should trigger only if all collected samples were above the threshold on 10m interval. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `003ef3a518`)	2023-10-25 14:39:48 +02:00
hagen1778	361a3dcc0c	deployment/alerts: make `RemoteWriteConnectionIsSaturated` expr readable Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `685f9c3c98`)	2023-10-25 14:39:47 +02:00
hagen1778	eb00a575c2	docs: mention key concepts as querying example See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5169 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-18 20:12:06 +02:00
hagen1778	46770409d9	dashboards/vmalert: respect job and instance filters in `No data errors` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `c2d252c045`)	2023-10-17 10:26:32 +02:00
hagen1778	d7bae2b78f	dashboards/vmalert: use `desc` sorting for tooltips on panels Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `edba9f6266`)	2023-10-17 10:26:32 +02:00
Aliaksandr Valialkin	767691b1a0	docs/Articles.md: add newly appeared articles about VictoriaMetrics at medium.com - https://sarthak-acoustic.medium.com/solving-metrics-at-scale-with-victoriametrics-ac9c306826c3 - https://medium.com/@seifeddinerajhi/victoriametrics-a-comprehensive-guide-comparing-it-to-prometheus-and-implementing-kubernetes-03eb8feb0cc2	2023-10-17 00:55:57 +02:00
Aliaksandr Valialkin	ace11318da	docs/CHANGELOG.md: document v1.93.6 LTS release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.6	2023-10-17 00:53:32 +02:00
Aliaksandr Valialkin	8642418e5a	app/vmselect: follow-up for `348c1bcec0`: cache static contents served from /select/tenantID/prometheus/vmui/static/...	2023-10-16 23:27:06 +02:00
Aliaksandr Valialkin	a6ad9208ef	docs/CHANGELOG.md: document v1.87.10 release See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.10	2023-10-16 23:25:55 +02:00
Aliaksandr Valialkin	627a4e9330	app/vmselect/promql: add labels_equal(q, "label1", "label2", ...) function This function returns q series, which have identical values for the listed labels "label1", "label2", ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148	2023-10-16 21:51:13 +02:00
Aliaksandr Valialkin	cdf62fb06b	docs/MetricsQL.md: typo fix after `bdb743c88d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071	2023-10-16 21:09:50 +02:00
Aliaksandr Valialkin	b2f9b9d634	app/vmselect/promql: add drop_empty_series() function for dropping empty series before performing additional calculations This can be useful in the following queries: drop_empty_series(temperature <= 30) default 40 This query drops temperature series with all the values bigger than 30 on the selected time range, while replacing gaps in the remaining series with 40. The query without drop_empty_series: (temperature <= 30) default 40 would leave all the temperature series with all the values bigger than 30 on the selected time range, and replace all their values with 40. This is not what could be epxected in some cases like here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071	2023-10-16 20:59:21 +02:00
Aliaksandr Valialkin	b28f904dfa	app/vmagent/remotewrite: move sas var initialization closer to the place where it is used This makes the code sligthtly easier to understand. This is a follow-up for `1d3d989be5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5170	2023-10-16 20:54:35 +02:00
hagen1778	1152c30430	app/vmagent/remotewrite: follow-up after `4f102ff945` `4f102ff945` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-16 20:54:35 +02:00
luosjde	c5bd3ff874	vmagent: fix streamaggr config reload bug https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5170 Authored-by: luoshaojun01 <luoshaojun01@baidu.com>	2023-10-16 20:54:35 +02:00
Aliaksandr Valialkin	2a0f77aaf7	lib/promscrape: add a link to https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets in descriptions for -promscrape.cluster.* command-line flags This should help users figuring out the purpose of -promscrape.cluster.* command-line flags	2023-10-16 14:47:38 +02:00
Aliaksandr Valialkin	fb1fcbefda	vendor: run `make vendor-update`	2023-10-16 14:33:16 +02:00
Aliaksandr Valialkin	92e79e835e	lib/promutils, app/vmalert-tool/unittest: move promutils.Duration.ParseTime() to app/vmalert-tool/unittest.durationToTime() The ParseTime() function looks strange, since it converts relative duration to absolute time since Unix Epoch. In most scenarios such a conversion is used by mistake. It is better to do not expose such a function for public use and hide it inside the package where it is needed, e.g. inside app/vmalert-tool/unittest. This is a follow-up for `dc28196237` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4789	2023-10-16 14:23:41 +02:00
Artem Navoiev	c8dfead6b1	docs/alert-tools add yaml output type for the correct displaying Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 14:12:34 +02:00
Haleygo	130e0ea5f0	vmalert-tool: implement unittest (#4789 ) 1. split package rule under /app/vmalert, expose needed objects 2. add vmalert-tool with unittest subcmd https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945	2023-10-16 14:12:06 +02:00
Aliaksandr Valialkin	3b79f98459	Revert "docs victorialogs use relative links" This reverts commit `3d7a77bf82`. Reason for revert: relative links do not work properly at GitHub code and at GitHub wiki. For example, the following page contains broken links before reverting this commit: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/VictoriaLogs/CHANGELOG.md It is always better to use absolute links thank relative links, since the page contents can be copy-n-pasted to other pages, which are located in vastly different directories, and all the links will remain working.	2023-10-16 14:03:21 +02:00
Aliaksandr Valialkin	64b503ba1c	docs/CaseStudies.md: typ fix: vmgent -> vmagent This is a follow-up for `f5c46b8176`	2023-10-16 14:02:58 +02:00
Aliaksandr Valialkin	d6f585ec0f	docs/vmbackup.md: clarify documentation about -deleteAllObjectVersions command-line flag Updates `2fc7e9f47e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121	2023-10-16 14:02:37 +02:00
Alexander Marshalov	1b4e7fcdb3	fixed error when creating a full backup using the `-origin` flag (#5180 ) * fixed error when creating a full backup using the `-origin` flag (#5144) * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-10-16 14:01:16 +02:00
Haleygo	93589ecccf	fix ingesting stale point, follow up `fe8cc573d1` (#5179 )	2023-10-16 14:00:39 +02:00
Github Actions	700d83db93	Automatic update operator docs from VictoriaMetrics/operator@79298bf (#5177 )	2023-10-16 13:56:49 +02:00
Aliaksandr Valialkin	4278b00a66	app/vmselect/promql: do not use unsafe conversion from bytes slice to string when storing a value by map key The assigned map key shouldn't change over time, otherwise the map won't work properly. This is a follow-up for `1f91f22b5f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087	2023-10-16 13:55:40 +02:00
Aliaksandr Valialkin	815e9bf892	app/{vmagent,vminsert}: follow-up for NewRelic data ingestion protocol support This is a follow-up for `f60c08a7bd` Changes: - Make sure all the urls related to NewRelic protocol start from /newrelic . Previously some urls were started from /api/v1/newrelic - Remove /api/v1 part from NewRelic urls, since it has no sense - Remove automatic transformation from CamelCase to snake_case for NewRelic labels and metric names, since it may complicate the transition from NewRelic to VictoriaMetrics. Preserve all the metric names and label names, so users could query metrics and labels by the same names which are used in NewRelic. The automatic transformation from CamelCase to snake_case can be added later as a special action for relabeling rules if needed. - Properly update per-tenant data ingestion stats at app/vmagent/newrelic/request_handler.go . Previously it was always zero. - Fix NewRelic urls in vmagent when multitenant data ingestion is enabled. Previously they were mistakenly started from `/`. - Document NewRelic data ingestion url at docs/Cluster-VictoriaMetrics.md - Remove superflouos memory allocations at lib/protoparser/newrelic - Improve tests at lib/protoparser/newrelic/* Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712	2023-10-16 13:55:04 +02:00
Aliaksandr Valialkin	8cdf57bfb1	docs/Single-server-VictoriaMetrics.md: add a link to the original issue, which describes how to run VictoriaMetrics as Windows service This is a follow-up for `cc7d5b7bab` The original issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781 contains up-to-date information on how to run VictoriaMetrics components as Windows service, plus it may contain additional information about this case such as https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781#issuecomment-1708092680 , so it is better to refer this issue from the docs.	2023-10-16 13:50:47 +02:00
Artem Navoiev	d2bf73bd07	docs fix broken links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:44:39 +02:00
Artem Navoiev	146c4fc494	docs fix broken links in operator Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:43:56 +02:00
Artem Navoiev	1cab088282	docs vlogs changelog fix broken link Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:43:34 +02:00
Artem Navoiev	998a4f72e0	docs fix 404 external links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:43:14 +02:00
Artem Navoiev	bb18168b83	docs fix bad links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:42:48 +02:00
Artem Navoiev	13a3daf700	docs victorialogs use relative links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:42:26 +02:00
Artem Navoiev	798e349bc8	docs/vlogs/dataingetion remoev unknown language from codeblock Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:27:18 +02:00
Artem Navoiev	c9a1fc73a6	docs/vmlogs fix aliases Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:26:11 +02:00
hagen1778	5f621b2fa1	docs: fix typo Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-16 13:25:05 +02:00
hagen1778	709a2bad66	docs: remove extra `/` in the end of the link Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-16 13:24:30 +02:00
Artem Navoiev	860a8136f1	docs remove (( from the link to fix them Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 13:20:50 +02:00
Zakhar Bessarab	3ba2be1657	deployment/logs-benchmark: add suite for Loki (#5165 ) * deployment/logs-benchmark: add suite for Loki Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/logs-benchmark: update go image to 1.21.3 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/logs-benchmark: split command to run elk and loki Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-16 12:53:56 +02:00
Aliaksandr Valialkin	4fd99f50af	docs/Cluster-VictoriaMetrics.md: follow-up after `f42c12e69a` Return back accidentally deleted text regarding KISS principles used in VictoriaMetrics development	2023-10-16 12:52:25 +02:00
Aliaksandr Valialkin	dfc67aedcb	lib/license: cleanups and prettifications for log messages and docs related to licensing - Make more clear the docs at docs/enterprise.md, so readers could figure out faster on how to obtain enterprise key and how to pass it to VictoriaMetrics Enterprise components. - Fix examples at docs/enterprise.md, which were referring to non-existing `-license-file` command-line flag. The `-licenseFile` command-line flag must be used instead. - Improve the description of `-license` command-line flags, so users could understand faster how to use them. - Improve the warning message, which is emitted when the deprecated -eula command-line flag is passed, so the user could figure out how to switch faster to -license command-line flags. - Disallow running VictoriaMetrics components with both -license and -licenseFile command-line flags. - Disallow running VictoriaMetrics components when -licensFile points to an empty file. - Consistently use the phrase "This flag is available only in Enterprise binaries" across all the enterprise-specific command-line flags. - Remove unneeded level of indirection for `noLicenseMessage` and `expiredMessage` string contants in order to improve code readability and maintainability. - Remove unneded `return` statements after `logger.Fatalf()` calls, since these calls exit the app and never return. - Make sure that the info log message about successful license verification is emitted when the license is verified successfully. Previously the error message could be logged when the license payload is invalid or if it misses some required features.	2023-10-16 12:51:37 +02:00
Artem Navoiev	0a0cd285db	docs: mention the free trial for enterprise Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `636d4ea196`)	2023-10-16 12:47:19 +02:00
Artem Navoiev	365cbf2047	fix title for managed qs guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-16 12:43:32 +02:00
Aliaksandr Valialkin	81289c8ab3	Revert "improve .gitignore" This reverts commit `a8345bb1b9` Reason for revert: VictoriaMetrics binaries are consistently created inside `bin` directory at the root of the repository when running `make <vm-app>` according to https://docs.victoriametrics.com/#how-to-build-from-sources If some dev environments create binaries inside random directories, then it is better to provide docs at https://docs.victoriametrics.com/#how-to-build-from-sources on how to setup these IDEs, so they consistently create binaries at bin/* directory at the root of the repository instead of trying to add random ignore rules inside .gitignore. As for the data directories created by VictoriaMetrics components, they may be created at random places too, so there is little sense in trying to add ignore rules for all these directories inside .gitignore. It is better to document that the built binaries must be consistently started from the repository root, so data directories are created at the repository root. The .gitignore already contains rule for blocking common data directories, which can be created by VictoriaMetrics components at the repository root.	2023-10-16 12:42:46 +02:00
Aliaksandr Valialkin	10420603b2	docs/Single-server-VictoriaMetrics.md: remove question mark added by accident in the commit `3d5d62e38a`	2023-10-16 12:41:53 +02:00
Aliaksandr Valialkin	b86bec8109	app/vmui: small UX enhancements - Reduce vertical space usage, so more information is available on the screen without the need to scroll. - Show information for lines with higher values at the top of the legend under the graph. This should simplify graph analysis when it contains many lines.	2023-10-16 12:39:32 +02:00
Aliaksandr Valialkin	ca35e0d66a	deployment/docker: update Go builder from Go1.21.1 to Go1.21.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved and https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved	2023-10-16 12:38:14 +02:00
Aliaksandr Valialkin	348c1bcec0	app/{vmselect,vlselect}: enable caching of static contents from /vmui/static/ folder at client side This should improve repated VMUI page load times on slow networks See https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/	2023-10-16 12:36:34 +02:00
Zakhar Bessarab	a353fb7e00	docs/vmbackupmaanger: reference permanent objects deletion (#5157 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-16 12:14:43 +02:00
hagen1778	b57e8b1bb9	dasbhoards: fix vminsert/vmstorage/vmselect metrics filtering Fix vminsert/vmstorage/vmselect metrics filtering when dashboard is used to display data from many sub-clusters with unique job names. Before, only one specific job could have been accounted for component-specific panels, instead of all available jobs for the component. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-16 12:13:01 +02:00
Dmytro Kozlov	d0df4f7c7f	docs/managed-victoriametrics: update documentation (#5135 ) Co-authored-by: Ivan Yatskevich <ivan@yatskevich.com>	2023-10-16 02:02:42 +02:00
Roman Khavronenko	8b2c30c51b	lib/vmselect: bump maxSearchQuerySize to 5MB (#5158 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154#issuecomment-1757216612 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-11 12:25:54 +02:00
Zakhar Bessarab	8b42a1733c	lib/backup: add `-deleteAllObjectVersions` command-line flag (#5147 ) New flag enforces removal of all versions of the object in remote object storage. See: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121 - https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages (cherry picked from commit `2fc7e9f47e`)	2023-10-10 14:14:21 +02:00
Yury Molodov	65a9f3da03	vmui: transfer Top Queries time interval #5097 (#5145 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5097 (cherry picked from commit `6dc5306c9b`)	2023-10-10 14:14:20 +02:00
Nikolay	4a50e9400c	app/vmselect: reduce lock contention for heavy aggregation requests (#5119 ) reduce lock contention for heavy aggregation requests previously lock contetion may happen on machine with big number of CPU due to enabled string interning. sync.Map was a choke point for all aggregation requests. Now instead of interning, new string is created. It may increase CPU and memory usage for some cases. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087	2023-10-10 13:44:02 +02:00
Haleygo	b52f1d1f0a	vmalert: add `evalAlignment` for rule group and fix evalutaion timstamp (#5066 ) * vmalert: add `query_time_alignment` for rule group 1. add `eval_alignment` attribute for group which by default is true. So group rule query stamp will be aligned with interval and propagated to ALERT metrics and the messages for alertmanager; 2. deprecate `datasource.queryTimeAlignment` flag. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049 (cherry picked from commit `2aa0f5fc41`)	2023-10-10 12:45:37 +02:00
Dmytro Kozlov	1cc6cd3d4f	app/vmalert: hide sensetive info in the vmalert (#5059 ) Strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044 (cherry picked from commit `244c887825`)	2023-10-10 12:45:36 +02:00
hagen1778	a906837403	docs: follow-up after `636d4ea196` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b57756734e`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-10 12:45:36 +02:00
Yury Molodov	29487700d9	vmui: enhancement of autocomplete feature (#5051 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4993 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006 (cherry picked from commit `c5044cdba9`)	2023-10-10 12:45:08 +02:00
Artem Navoiev	d8c8a66c79	docs: mention the free trial for enterprise Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `636d4ea196`)	2023-10-10 12:45:08 +02:00
Zakhar Bessarab	a92da26108	docs/enterprise: fix rendering of example alert (#5137 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `7183621d84`)	2023-10-10 12:45:07 +02:00
Artem Navoiev	eea2e476cb	docs: fix markdown for k8s managed guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `dc79b25771`)	2023-10-10 12:45:07 +02:00
hagen1778	9decf32799	docs: add YT video to "How to reduce expenses on monitoring" Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `cb13ce9147`)	2023-10-10 12:45:07 +02:00
Artem Navoiev	65b2a0ce60	docs: update the license flags description Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `b3cc22b159`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-10 12:45:06 +02:00
Artem Navoiev	5b7fb78a6e	improve .gitignore Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `a8345bb1b9`)	2023-10-10 12:44:40 +02:00
hagen1778	a41c2aeafb	app/(vminsert\|vmagent): fix label names for newrelic metrics Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `7cf615a73b`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-05 15:08:07 +02:00
Dmytro Kozlov	a89cf4ce12	app/(vminsert\|vmagent): add support for new relic infrastructure agent (#4712 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f60c08a7bd`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-05 15:06:05 +02:00
Zakhar Bessarab	99582f19d2	docs: add example service configuration for windows (#5129 ) Use service configuration example from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781#issuecomment-1424671624 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `cc7d5b7bab`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-05 15:00:50 +02:00
Denys Holius	f42c12e69a	docs: update social links (cherry picked from commit `3d5d62e38a`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-05 15:00:12 +02:00
Github Actions	c488f225ec	Automatic update operator docs from VictoriaMetrics/operator@26d1b3b (#5130 ) (cherry picked from commit `b576acff5e`)	2023-10-05 14:59:51 +02:00
Aliaksandr Valialkin	c3908350ea	docs/VictoriaLogs/Roadmap.md: add integration with Grafana	2023-10-05 11:41:01 +02:00
Zakhar Bessarab	07586c1d8f	docs/enterprise: add info about license flags (#5085 ) * docs/enterprise: add info about license flags Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/enterprise: add more examples on how to run enterprise components Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/enterprise: address review feedback Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/enterprise: specify release version Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/enterprise: fix image tag in example Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/enterprise: add docker-compose example, fix secret name Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-05 11:39:22 +02:00
Artem Navoiev	5c87066f8c	docs: victorialogs change the doc ordering.2 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-05 11:37:01 +02:00
Artem Navoiev	8f96709219	docs: victorialogs change the doc ordering Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-05 11:36:27 +02:00
Github Actions	380a5d2c9f	Automatic update operator docs from VictoriaMetrics/operator@5da4bf6 (#5127 )	2023-10-05 11:35:16 +02:00
Aliaksandr Valialkin	7ccefffe27	docs/VictoriaLogs/CHANGELOG.md: add release date for v0.4.1-victorialogs	2023-10-04 17:57:25 +02:00
Github Actions	e20f969ed3	Automatic update operator docs from VictoriaMetrics/operator@9fb79f4 (#5124 )	2023-10-04 17:57:25 +02:00
Aliaksandr Valialkin	e275be05b0	docs/vmbackup.md: mention that -filestream.disableFadvise command-line flag can be used for reducing CPU usage on systems with big number of CPU cores Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120 This is a follow-up for `75dd7b30ba`	2023-10-04 16:38:53 +02:00
Zakhar Bessarab	dbf0aa4b87	docs/vmbackupmanager: add missing step for restore flow in k8s (#5116 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-04 16:38:53 +02:00
Aliaksandr Valialkin	4e5d3d38f8	lib/filestream: add `-filestream.disableFadvise` syscall for unconditional disabling of `fadvise` syscall This may be needed in rare cases when performing backups on systems with big number of CPU cores and big value passed to -concurrency command-line flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120	2023-10-04 16:30:19 +02:00
Artem Navoiev	7a5a40e031	docs: fix wieghts for logs folders Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-04 16:29:31 +02:00
Artem Navoiev	1a2af79047	docs: remove id clashes for pages, properly cover victorialogs changelog page (#5123 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-04 16:28:52 +02:00
Artem Navoiev	12b0e3e5cb	add hugo front matter for operator docs (#5122 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-10-04 16:28:23 +02:00
Aliaksandr Valialkin	65e7a7cabb	deployment/docker: update VictoriaLogs from v0.4.0-victorialogs to v0.4.1-victorialogs See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v041	2023-10-04 16:27:39 +02:00
f41gh7	10569b9d80	cut v0.4.1-victorialogs release	2023-10-04 14:30:22 +02:00
Aliaksandr Valialkin	776174f1cd	docs/Single-server-VictoriaMetrics.md: follow-up for `612ec2df6b` Move the description of JSON line format into a separate chapter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083	2023-10-03 17:53:15 +02:00
hagen1778	297f63a01e	alerting: account for `vmauth` component for alerts `ServiceDown` and `TooManyRestarts` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-03 17:52:43 +02:00
Alexander Marshalov	44b51808ec	hide deprecated docs from menu (#5095 )	2023-10-03 17:52:15 +02:00
Zakhar Bessarab	9f5e6fe0eb	docs/victorialogs: changelog followup for `650d72f3` (#5114 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-03 17:50:49 +02:00
Zakhar Bessarab	85b604d414	lib/logstorage: fix free space check (#5113 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-03 17:50:16 +02:00
hagen1778	c0b478d47f	docs: add more details explaining JSON line format See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-03 17:48:33 +02:00
Aliaksandr Valialkin	dc71db9470	deployment: update VictoriaLogs from v0.3.0-victorialogs to v0.4.0-victorialogs See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v040	2023-10-03 02:34:28 +02:00
Aliaksandr Valialkin	ff471d9f2c	docs/VictoriaLogs/CHANGELOG.md: cut v0.4.0-victorialogs	2023-10-03 01:46:42 +02:00
Aliaksandr Valialkin	fcd43f11f5	deployment: update VictoriaMetrics from v1.93.5 to v1.94.0 See https://docs.victoriametrics.com/CHANGELOG.html#v1940	2023-10-03 01:40:21 +02:00
Aliaksandr Valialkin	cf93d27bca	docs/CHANGELOG.md: cut v1.94.0	2023-10-02 22:33:40 +02:00
Aliaksandr Valialkin	f0ff9c6f8c	docs/Articles.md: add a video from GopherCon 2023 for `Writing a TSDB from scratch: performance optimization` talk	2023-10-02 22:24:48 +02:00
Aliaksandr Valialkin	47741cd935	deployment/docker: update Alpine from 3.18.3 to 3.18.4 See https://alpinelinux.org/posts/Alpine-3.18.4-released.html	2023-10-02 21:54:57 +02:00
Aliaksandr Valialkin	cd1169618f	vendor: run `make vendor-update`	2023-10-02 21:53:35 +02:00
Aliaksandr Valialkin	b5812e2457	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update`	2023-10-02 21:44:21 +02:00
Yury Molodov	f5521ce4d4	vmui: add storage for query history (#5022 ) * vmui: add storage for query history * docs/vmui: add storage for query history	2023-10-02 21:41:32 +02:00
Roman Khavronenko	1f2cb594d9	lib/promscrape: make concurrency control optional (#5073 ) * lib/promscrape: make concurrency control optional Before, `-maxConcurrentInserts` was limiting all calls to `promscrape.Parse` function: during ingestion and scraping. This behavior is incorrect. Cmd-line flag `-maxConcurrentInserts` should have effect onl on ingestion. Since both pipelines use the same `promscrape.Parse` function, we extend it to make concurrency limiter optional. So caller can decide whether concurrency should be limited or not. This commit makes `c53b5788b4` obsolete. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Revert "dashboards: move `Concurrent inserts` panel to Troubleshooting section" This reverts commit `c53b5788b4`. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-02 21:34:41 +02:00
Yury Molodov	40973d37da	vmui: improve the appearance of the trace (#5091 )	2023-10-02 21:24:51 +02:00
Dmytro Kozlov	d8d003af24	app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes (#5106 ) * app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes * app/vmagent: fix CHANGELOG.md description * wip * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-10-02 21:18:47 +02:00
Aliaksandr Valialkin	d80ccf52a0	Revert "lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total` (#5074 )" This reverts commit `74301cdbf5`. Reason for revert: vmagent already provides better approach for detecting slow scrape targets via the following query: scrape_duration_seconds / scrape_timeout_seconds > 1 This query depends on automatically generated per-target metrics. See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5074	2023-10-02 21:08:13 +02:00
Aliaksandr Valialkin	5fd79f47f1	app/vmselect/promql: follow-up for `896c85a4a4` - Clarify the description of the change at docs/CHANGELOG.md - Make sure that bitmap_*(X, NaN) returns NaN Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5021	2023-10-02 21:07:46 +02:00
Aliaksandr Valialkin	7bb5f75a2a	lib/logstorage: follow-up for `94627113db` - Move uniqueFields from rows to blockStreamMerger struct. This allows localizing all the references to uniqueFields inside blockStreamMerger.mustWriteBlock(), which should improve readability and maintainability of the code. - Remove logging of the event when blocks cannot be merged because they contain more than maxColumnsPerBlock, since the provided logging didn't provide the solution for the issue with too many columns. I couldn't figure out the proper solution, which could be helpful for end user, so decided to remove the logging until we find the solution. This commit also contains the following additional changes: - It truncates field names longer than 128 chars during logs ingestion. This should prevent from ingesting bogus field names. This also should prevent from too big columnsHeader blocks, which could negatively affect search query performance, since columnsHeader is read on every scan of the corresponding data block. - It limits the maximum length of const column value to 256. Longer values are stored in an ordinary columns. This helps limiting the size of columnsHeader blocks and improving search query performance by avoiding reading too long const columns on every scan of the corresponding data block. - It deduplicates columns with identical names during data ingestion and background merging. Previously it was possible to pass columns with duplicate names to block.mustInitFromRows(), and they were stored as is in the block. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4969	2023-10-02 21:06:49 +02:00
Dmytro Kozlov	e7b35455bd	docs: add clarification of the retention filter usage (#5103 ) docs: add clarification of the retention filter usage Updated documentation regarding retention filter usage if duration is set lower than `-retentionPeriod` flag value. Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-10-02 21:05:54 +02:00
Roman Khavronenko	0df0b0f29e	lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total` (#5074 ) * lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total` add metric `vm_promscrape_scrapes_skipped_total`to show whether vmagent skips the scrapes. This could happen if vmagent is overloaded or target is responding too slow for configured `scrape_interval`. The follow-up commit should add a corresponding alerting rule and panel to vmagent dashboard. Signed-off-by: hagen1778 <roman@victoriametrics.com> * deployment/docker: add `TooManyScrapeSkips` alerting rule for vmagent Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards: add panels `Scrape duration 0.99 quantile` and `Skipped scrapes` to vmagent dashboard Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-02 20:38:23 +02:00
Aliaksandr Valialkin	120f3bc467	lib/logstorage: follow-up for `8a23d08c21` - Compare the actual free disk space to the value provided via -storage.minFreeDiskSpaceBytes directly inside the Storage.IsReadOnly(). This should work fast in most cases. This simplifies the logic at lib/storage. - Do not take into account -storage.minFreeDiskSpaceBytes during background merges, since it results in uncontrolled growth of small parts when the free disk space approaches -storage.minFreeDiskSpaceBytes. The background merge logic uses another mechanism for determining whether there is enough disk space for the merge - it reserves the needed disk space before the merge and releases it after the merge. This prevents from out of disk space errors during background merge. - Properly handle corner cases for flushing in-memory data to disk when the storage enters read-only mode. This is better than losing the in-memory data. - Return back Storage.MustAddRows() instead of Storage.AddRows(), since the only case when AddRows() can return error is when the storage is in read-only mode. This case must be handled by the caller by calling Storage.IsReadOnly() before adding rows to the storage. This simplifies the code a bit, since the caller of Storage.MustAddRows() shouldn't handle errors returned by Storage.AddRows(). - Properly store parsed logs to Storage if parts of the request contain invalid log lines. Previously the parsed logs could be lost in this case. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4945	2023-10-02 20:38:00 +02:00
Aliaksandr Valialkin	cbbdf9cdf5	lib/logstorage: run up to GOMAXPROCS flushers of old in-memory parts to disk One flusher isn't enough under high data ingestion rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775	2023-10-02 20:36:53 +02:00
Github Actions	025a53dcb6	Automatic update operator docs from VictoriaMetrics/operator@44bdc27 (#5104 )	2023-10-02 20:36:21 +02:00
Github Actions	835393a59c	Automatic update operator docs from VictoriaMetrics/operator@c7125bd (#5102 )	2023-10-02 20:36:21 +02:00
hagen1778	25a006099d	app/vlinsert/loki: make fmt Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-02 20:35:45 +02:00
Aliaksandr Valialkin	78e9cda4b1	lib/logstorage: assist merging in-memory parts at data ingestion path if their number starts exceeding maxInmemoryPartsPerPartition This is a follow-up for `9310e9f584` , which removed data ingestion pacing. This can result in uncontrolled growth of in-memory parts under high data ingestion rate, which, in turn, can result in unbounded RAM usage, OOM crashes and slow query performance. While at it, consistently reset isInMerge field for parts passed to mergeParts() before returning from this function. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4828	2023-10-02 20:35:20 +02:00
Aliaksandr Valialkin	f55d114785	lib/{mergeset,storage}: consistently reset isInMerge field in parts passed to mergeParts() before returning from the function While at it consistently check that the isInMerge field is set in all the parts passed to mergeParts()	2023-10-02 20:34:52 +02:00
Aliaksandr Valialkin	c3ece6d38e	docs/VictoriaLogs/CHANGELOG.md: remove duplicate lines about vl_http_request_duration_seconds metric This is a follow-up after `8a23d08c21` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4945	2023-10-02 20:34:19 +02:00
Aliaksandr Valialkin	8b1d6b995e	lib/{mergeset,storage}: perform at most one assisted merge per each call to addRows/addItems This should reduce tail latency during data ingestion. This shouldn't slow down data ingestion in the worst case, since assisted merges are spread among distinct addRows/addItems calls after this change.	2023-10-02 20:33:51 +02:00
Aliaksandr Valialkin	4c0402f118	docs/Single-server-VictoriaMetrics.md: refer to `active queries` and `top queries` pages at VMUI instead of refering to the corresponding HTTP endpoints "Active queries" and "Top queries" pages at VMUI are user-friendly than the corresponding HTTP endpoints	2023-10-02 20:33:10 +02:00
Aliaksandr Valialkin	5f1492d978	docs/vmalert.md: refer to -evaluationInterval command-line flag instead of evaluation_interval option, which isnt supported by vmalert This is follow-up for `5c42c1218a`	2023-10-02 20:32:02 +02:00
Aliaksandr Valialkin	ac418281da	docs/Troubleshooting.md: describe how to optimize SLI/SLO queries with long lookbehind windows	2023-10-02 20:29:37 +02:00
Aliaksandr Valialkin	b5f9a6a5c6	docs/CHANGELOG.md: follow-up for `f0e33700fc` Mention that the statistic inaccuracy is related to cardinality explorer	2023-10-02 20:29:07 +02:00
Aliaksandr Valialkin	3db9db356d	deployment/docker/docker-compose-cluster.yml: follow-up for `4d1b572f46` Grafana and vmalert now depend on vmauth instead of individual vmselect nodes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5057	2023-10-02 20:28:06 +02:00
Aliaksandr Valialkin	b366a22018	deployment: update VictoriaMetrics version from v1.93.4 to v1.93.5 See https://docs.victoriametrics.com/CHANGELOG.html#v1935	2023-10-02 20:16:54 +02:00
Dmytro Kozlov	90b189dab8	app/vmselect: fix bitmap_*() functions behavior (#5021 ) Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996 Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-10-02 20:13:27 +02:00
Aliaksandr Valialkin	6e613cb8e8	docs/Cluster-VictoriaMetrics.md: increase the minimum supported version of Go builder from 1.18 to 1.20 See the related commit `3da493ff62`	2023-10-02 19:33:41 +02:00
Zakhar Bessarab	876bce5a57	lib/logstorage: prevent from panic during background merge (#4969 ) * lib/logstorage: prevent from panic during background merge Fixes panic during background merge when resulting block would contain more columns than maxColumnsPerBlock. Buffered data will be flushed and replaced by the next block. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/logstorage: clarify field description and comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-02 19:29:31 +02:00
Zakhar Bessarab	dfdada055c	lib/logstorage: switch to read-only mode when running out of disk space (#4945 ) * lib/logstorage: switch to read-only mode when running out of disk space Added support of `--storage.minFreeDiskSpaceBytes` command-line flag to allow graceful handling of running out of disk space at `--storageDataPath`. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/logstorage: fix error handling logic during merge Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/logstorage: fix log level Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-10-02 17:09:57 +02:00
Zakhar Bessarab	53268ebc66	lib/logstorage/datadb: remove parts merge cond (#4828 ) It was added in order to limit number of goroutines performing assisted merges during ingestion. It turned out that blocking ingestion goroutines lower ingestion performance and limits overall ingestion around 40k items per seconds because of lock contention. Removing parts merge sync.Cond allows to remove lock contention at write path and significantly improves write performance. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-02 17:09:12 +02:00
Dmytro Kozlov	10371eac60	vmui: update information about tsdb usage in cluster version (#5004 ) * vmui: update information about tsdb usage in cluster version * vmui: cleanup * vmui: add CHANGELOG.md * vmui: cleanup * vmui: update logic, move information to the visible place * app/vmui: remove values fetch, update documentation for cardinality explorer * app/vmui: update CHANGELOG.md	2023-10-01 21:30:44 +02:00
Zakhar Bessarab	ff88e53e01	doc: address review feedback Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-01 21:27:49 +02:00
Zakhar Bessarab	9f6704d6cc	doc: mention InfluxDB v2 HTTP API support Address: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5076 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-10-01 21:26:44 +02:00
Github Actions	18cc977956	Automatic update operator docs from VictoriaMetrics/operator@958ce2b (#5070 )	2023-10-01 21:25:54 +02:00
hagen1778	d0641d6ea2	dashboards: move `Concurrent inserts` panel to Troubleshooting section Moved because this panel is related to both: scraped and ingested data. Before, it could have give a misleading impression that it is related to ingested metrics only. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-10-01 21:25:25 +02:00
Roman Khavronenko	d39c8525e2	Docker add vmauth (#5057 ) * docker-compose: add vmauth to cluster env vmauth acts as a balancer and used as an example of how to interconnect VM components via vmauth. Signed-off-by: hagen1778 <roman@victoriametrics.com> * docker-compose: add vmauth to cluster env vmauth acts as a balancer and used as an example of how to interconnect VM components via vmauth. Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-10-01 21:24:01 +02:00
Aliaksandr Valialkin	15645c8a94	app/vmagent: follow-up for `cfef814750` - Properly handle /insert/multitenant/api/put url for opentsdb handler at vmagent - Document that the bug has been introduced in v1.93.2 at docs/CHANGELOG.md - Add a link to multitenant url docs in bugfix description Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910	2023-10-01 20:52:29 +02:00
Alexander Marshalov	cfef814750	fixed ingestion via multitenant url for opentsdbhttp (#5061 ) (#5063 )	2023-09-26 10:47:49 +02:00
Aliaksandr Valialkin	e341128096	docs/vmagent.md: make VictoriaMetrics remove_write protocol more visible by mentioning it at the top of the page	2023-09-25 17:42:25 +02:00
Aliaksandr Valialkin	9ae92ff2ee	lib/storage: remove unused atomicSetBool function after `717c53af27`	2023-09-25 17:37:45 +02:00
Aliaksandr Valialkin	308134970f	docs: run `make docs-sync` after `8e722e10ee`	2023-09-25 17:35:34 +02:00
Aliaksandr Valialkin	f6b35a715d	docs/CaseStudies.md: add Criteo case study This is a follow-up for `bdbe616408` See https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61	2023-09-25 17:34:47 +02:00
Aliaksandr Valialkin	60fe63df07	lib/storage: make it clear that the number of big merge workers always equals to 4 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4915#issuecomment-1733922830	2023-09-25 17:17:40 +02:00
Aliaksandr Valialkin	a421db5977	lib/storage: stop exposing vm_merge_need_free_disk_space metric This metric confuses users and has no any useful information. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686#issuecomment-1733844128	2023-09-25 17:00:14 +02:00
Aliaksandr Valialkin	538dc6058d	app/vmselect/promql: run `make fmt` after `3b9605dba5`	2023-09-25 16:15:58 +02:00
Aliaksandr Valialkin	b43ff80d21	app/vmselect/promql: do not sort `q1 or q2` results This makes sure that `q2` series are returned after `q1` series in the same way as Prometheus does See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763	2023-09-25 16:15:02 +02:00
Aliaksandr Valialkin	c954019e43	app/vmselect/promql: completely substitute median_over_time() WITH template with regular median_over_time() rollup function This is a follow-up for `34d7a670d0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034	2023-09-25 15:31:25 +02:00
Zakhar Bessarab	fd6ca57c14	app/vmselect/promql: add implementation of median_over_time for rollup functions list (#5042 ) `median_over_time` is handled by predefined WITH template in MetricsQL library which translates it to `quantile_over_time(0.5)` This makes it impossble to use `median_over_time` as a usual rollup function for `aggr_over_time`. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-25 15:31:25 +02:00
Roman Khavronenko	23131f932a	docs/changelog: add link to sandbox (#5050 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-25 15:13:30 +02:00
Github Actions	d9754ca44b	Automatic update operator docs from VictoriaMetrics/operator@587ea54 (#5054 )	2023-09-25 15:12:44 +02:00
hagen1778	1ffc23af1b	docs/articles: add link to "How to reduce expenses on monitoring" slides Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-25 15:03:26 +02:00
Aliaksandr Valialkin	36d26b69aa	docs/Cluster-VictoriaMetrics.md: update `-help` output for enterprise components	2023-09-22 13:51:18 +02:00
Zakhar Bessarab	0be8960875	lib/promscrape/discovery/kubernetes: supress context.Cancelled error in logs (#5048 ) lib/promscrape/discovery/kubernetes: supress context.Cancelled error in logs It is possible that context.Cancelled will appear after k8s watcher was closed due to reload(see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850). Logging an error misinforms user and looks like vmagent discovery will stop working even though this does not affect discovery. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `8d99c12a7d`)	2023-09-22 13:02:57 +02:00
Zakhar Bessarab	86eaf6906b	docs/vmbackup: update docs for different authentication options, add examples (#5046 ) Updates: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5023 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-22 11:42:34 +02:00
Github Actions	391b857eb4	Automatic update operator docs from VictoriaMetrics/operator@9d65e09 (#5040 )	2023-09-22 11:41:33 +02:00
Aliaksandr Valialkin	281eb0c377	lib/storage: log fatal error inside searchMetricName() instead of propagating it to the caller This simplifies the code a bit at searchMetricName() and searchMetricNameWithCache() call sites This is a result of investigating https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4972	2023-09-22 11:37:55 +02:00
Zakhar Bessarab	e216592378	lib/backup: fix issue with inconsistent copying of appliedRetention.txt (#5027 ) * lib/backup: fix issue with inconsistent copying of appliedRetention.txt appliedRetention.txt can be modified in place, so it should be always copied just the same as parts.json Updates: https://github.com/victoriaMetrics/victoriaMetrics/issues/5005 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: add changelog entry for appliedRetention.txt copying fix Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-21 11:26:13 +02:00
Aliaksandr Valialkin	11ebcf1f9b	app/vmauth: consistently use '%w' for formatting errors in fmt.Errorf()	2023-09-21 11:05:26 +02:00
Roman Khavronenko	c9f121e694	app/vmauth: update config reload routine (#5019 ) * expose metrics `vmauth_config_last_reload_` for tracking the state of config reloads, similarly to vmagent/vmalert components. do not print logs like `SIGHUP received...` once per configured `-configCheckInterval` cmd-line flag. This log will be printed only if config reload was invoked manually. * prevent configuration reloading if there were no changes in config. This improves memory usage when `-configCheckInterval` cmd-line flag is configured and config has extensive list of regexp expressions requiring additional memory on parsing. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-21 11:05:26 +02:00
hagen1778	f2195cb914	dashboards/victoriametrics: account for instance filter in annotations Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-21 09:36:35 +02:00
Zakhar Bessarab	47d9e82b52	lib/storage/partition: add check to ensure parts exist on disk (#5017 ) * lib/storage/partition: add check to ensure parts exist on disk If part exists in parts.json but is missing on disk there will be a misleading error similar to "unexpected number of substrings in the part name". This change forces verification of part existence and throws a correct error in case it is missing on disk. Such issue can be result of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005 or disk corruption. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/partition: use filepath.Join instead of string concatenation Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage/partition: add action points for error message Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * all: add a check for missing part in lib/mergeset and lib/logstorage --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-19 11:18:21 +02:00
Aliaksandr Valialkin	919f7d2f7e	docs/CHANGELOG.md: publish changes for v1.93.5	2023-09-19 10:50:36 +02:00
hagen1778	e85c28f05c	docs/articles: add https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-19 10:34:08 +02:00
hagen1778	91601da2c2	build(deps): revert version change for codecov/codecov-action from 4 to 3 https://github.com/codecov/codecov-action/issues/1089 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-19 10:34:08 +02:00
Aliaksandr Valialkin	ddf88dddc0	docs/CHANGELOG.md: clarify the description of bugfixes at `f7dda12b4d` and `b6ad581b45` This is a follow-up for `8b01bc4a5c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4999 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5009	2023-09-19 00:45:59 +02:00
Aliaksandr Valialkin	cc8f2bee0d	app/vlinsert: follow-up for `d570763c91` - Switch from summary to histogram for vl_http_request_duration_seconds metric. This allows calculating request duration quantiles across multiple hosts via histogram_quantile(0.99, sum(vl_http_request_duration_seconds_bucket) by (vmrange)). - Take into account only successfully processed data ingestion requests when updating vl_http_request_duration_seconds histogram. Failed requests are ignored, since they may significantly skew measurements. - Clarify the description of the change at docs/VictoriaLogs/CHANGELOG.md. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4934	2023-09-19 00:45:28 +02:00
Aliaksandr Valialkin	de2b3ff9b0	lib/promscrape/discovery/kubernetes: follow-up after `03fece44e0` - Properly update vm_promscrape_discovery_kubernetes_url_watchers and vm_promscrape_discovery_kubernetes_group_watchers metrics after config changes - Properly stop goroutine responsible for recreating scrapeWorks after the corresponding urlWatcher is stopped - Log the event when urlWatcher is stopped in order to simplify debugging Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4861	2023-09-19 00:44:21 +02:00
Aliaksandr Valialkin	705b31c351	lib/promscrape/discovery/kubernetes: wait for 10 seconds before checking whether the urlWatcher must be stopped This should prevent from excess urlWatcher churn on config reload, since it leads to removal of all the apiWatchers before creating new apiWatchers. So, every config reload would lead to stopping of all the previous urlWatchers and starting new urlWatchers. The new logic gives 10 seconds for config reload before stopping unused urlWatchers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4861	2023-09-19 00:43:26 +02:00
Aliaksandr Valialkin	fe24523e19	lib/promscrape/discovery/kubernetes: follow-up after `eeb862f3ff` - Move the bugfix description to the correct place in docs/CHANGELOG.md - Prevent from logging of 'context canceled' errors after the url watcher is stopped, since these errors are expected and may confuse users. - Remove unused urlWatcher.refCount field. - Remove unused urlWatcher.close() method. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850	2023-09-19 00:41:29 +02:00
Aliaksandr Valialkin	d0c103ad05	lib/backup: properly copy parts.json files inside indexdb directory additional to data directory This is a follow-up for `264ffe3fa1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5006	2023-09-19 00:38:31 +02:00
Aliaksandr Valialkin	c0b8143daf	lib/backup/common: consistently use canonical path with / directory separators at Part.Path Previously Part.Path could contain `\` directory separators on Windows OS, which could result in incorrect filepaths generation when making backups at object storage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704 This is a follow-up for `f2df8ad480`	2023-09-19 00:36:32 +02:00
Aliaksandr Valialkin	9e50ea6cce	app/vlinsert/insertutils: cosmetic changes after `8d3e574c31`	2023-09-19 00:35:51 +02:00
dependabot[bot]	1870a5cac0	build(deps): bump codecov/codecov-action from 3 to 4 (#5011 ) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/v3...v4) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-09-19 00:35:10 +02:00
Nikolay	e5b62b0f62	docs: reflect recent changes at change logs (#5015 )	2023-09-19 00:24:28 +02:00
crossoverJie	fb13887573	app/vlinsert: Add vl_http_request_duration_seconds metrics (#4934 )	2023-09-19 00:10:24 +02:00
Zakhar Bessarab	55d25fb844	lib/promscrape/discovery/kubernetes: fix leaking api watcher (#4861 ) * lib/promscrape/discovery/kubernetes: fix leaking api watcher goroutine which was polling k8s API had no execution control. This leaded to leaking goroutines during config reload. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/kubernetes: use reference counting for urlWatcher cleanup Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/kubernetes: remove waitgroup sync for goroutines polling API server This is unnecessary since context will is cancelled and new requests will not be sent. Also, using waitgroup will increase time required to perform reload which might result in missed scrapes. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/kubernetes: clarify comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Apply suggestions from code review * lib/promscrape/discovery/kubernetes: address review feedback Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-09-18 17:13:55 +02:00
Konstantin	c1a8a2d54c	app/vmselect: return +Inf as null in graphite render api (#5009 ) Signed-off-by: Konstantin Kulikov <k.kulikov2@gmail.com>	2023-09-18 16:41:39 +02:00
faceair	609c76eec9	lib/storage: remove ForceMergeAllParts internal loop (#4999 ) Signed-off-by: faceair <git@faceair.me>	2023-09-18 16:35:37 +02:00
Zakhar Bessarab	aa583f0b9a	lib/backup: force copying of parts.json (#5006 ) * lib/backup: force copying of parts.json Copying of parts.json is required because `part.key()` comparison can create same key value for files with different contents. This will result in inconsistent backup being created or restored. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/backup: ensure parts.json is only copied once Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-09-18 16:18:03 +02:00
Zakhar Bessarab	b0c8f7f718	app/vlinsert: add flag to limit amount of fields per line (#4976 ) Adding limit on ingestion allows to avoid issues like this one https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762 Such issues are often caused by misconfigurtion on log persing/ingestion side and preventing such rows from being ingested allows to avoid performance implications created by storing such log rows. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-18 12:00:02 +02:00
Zakhar Bessarab	eefdc60c88	docs: add changelog entry for downsampling.period and dedup.minScrapeInterval verification (#5000 ) * docs: add changelog entry for downsampling.period and dedup.minScrapeInterval verification - added changelog entry - documented requirements for dedup.minScrapeInterval and downsampling.period being multiples of each other Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: `make docs-sync` Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-18 11:38:13 +02:00
Artem Navoiev	d435939109	add annotation to VictoriaLogs dashboards - restarts and version change (#5008 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-09-18 11:35:16 +02:00
Dmytro Kozlov	5477b52991	vmagent: add validation of MetricsQL functions (#4991 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-15 13:16:22 +02:00
Aliaksandr Valialkin	24cec79763	lib/storage: handle fatal errors inside indexSearch.getTSIDByMetricID() instead of returning them to the caller This simplifies the code a bit at caller side	2023-09-15 12:01:06 +02:00
Github Actions	bf30ded160	Automatic update operator docs from VictoriaMetrics/operator@49d245d (#5002 )	2023-09-15 11:58:54 +02:00
Github Actions	622bd073b6	Automatic update operator docs from VictoriaMetrics/operator@a9c54f4 (#4998 )	2023-09-15 11:58:54 +02:00
Aliaksandr Valialkin	f3eaeb8e5c	.github/workflow: remove automatic creation of pull request at github.com/VictoriaMetrics/ops repository on new tag This automation doesn't work as intended on LTS releases, bugfix releases and custom releases, since it assumes every new tag is related only to new release. Also the github.com/VictoriaMetrics/ops repository may contain manually set custom tags for VictoriaMetrics components (for example, for testing the latest bugfixes or features), which are overwritten by the generated pull request. The way to go is to manually update tags at github.com/VictoriaMetrics/ops repository when needed instead of trying to automate this process.	2023-09-11 23:52:55 +02:00
Alexander Marshalov	c57ce9a55a	Updates in release guide (#4956 ) new helm release flow and operator release flow Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-09-11 23:51:27 +02:00
Github Actions	9baaa47ef1	Automatic update operator docs from VictoriaMetrics/operator@21f7203 (#4988 )	2023-09-11 23:51:11 +02:00
Aliaksandr Valialkin	f9f3a871f7	docs/CHANGELOG.md: document v1.87.9	2023-09-10 21:41:56 +02:00
Aliaksandr Valialkin	1d9adc022d	deployment/docker: update VictoriaMetrics from v1.93.3 to v1.93.4 See https://docs.victoriametrics.com/CHANGELOG.html#v1934	2023-09-10 19:50:14 +02:00
Aliaksandr Valialkin	5f4ed9b45f	docs/CHANGELOG.md: document v1.93.4	2023-09-10 19:47:51 +02:00
Aliaksandr Valialkin	9c3a37597c	app/vmselect/netstorage: run `make fmt` after `58326dbf25`	2023-09-10 15:18:15 +02:00
Artem Navoiev	973c74df55	Update VL daashboard. Add Resource Section, add ds and job filters, a… (#4981 ) * Update VL daashboard. Add Resource Section, add ds and job filters, add metric collection in docker compose from victorialogs, fix networkigs usage in docker compose Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * add vl dashboard to docker compose Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * add vl dashboard to docker compose Signed-off-by: Artem Navoiev <tenmozes@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-09-10 15:05:19 +02:00
Github Actions	d2bc7e2160	Automatic update operator docs from VictoriaMetrics/operator@83c07ed (#4970 )	2023-09-08 23:33:42 +02:00
Aliaksandr Valialkin	a5a953fe1e	app/vmagent/remotewrite: fix data race when extra labels are added to samples before sending them to multiple remote storage systems See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4972	2023-09-08 23:26:40 +02:00
Aliaksandr Valialkin	23bb38ef80	app/vmauth: add ability to specify response status codes for retrying requests during load-balancing Response status codes for retrying can be specified via retry_status_codes list See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893	2023-09-08 23:26:36 +02:00
Zakhar Bessarab	b842a0cd25	docs: sync description for license flags (#4977 ) - update eula flag to add deprecation notice - add new license flags description Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-08 23:26:25 +02:00
Roman Khavronenko	548acce6b3	vmalert: correctly add duplicated params to the query (#4955 ) Fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in this commit `eccecdf177` starting from v1.91.1 https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6351d07da8`)	2023-09-08 09:33:45 +02:00
Aliaksandr Valialkin	3257fcfd8d	app/vmauth: retry requests at other backends on 5xx response status codes This should allow implementing high availability scheme described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792#issuecomment-1674338561 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893	2023-09-08 00:47:57 +02:00
Aliaksandr Valialkin	58326dbf25	app/vmselect: return 503 status code when partial responses are denied and some of vmstorage nodes are temporarily unavailable This should help detecting this case and automatic retrying the query at healthy cluster replica in another availability zone. This commit is needed as a preparation for automatic query retry at another backend at vmauth on 5xx errors as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792#issuecomment-1674338561	2023-09-07 16:07:06 +02:00
Aliaksandr Valialkin	4af79504f6	lib/logger: increase the maximum log arg size from 200 to 500 The 200 chars limit has been appeared too small for typical log messages emitted by VictoriaMetrics components This is a follow-up for `87fea7d8ac`	2023-09-07 15:58:57 +02:00
Aliaksandr Valialkin	dfe44c917c	lib/logger: limit the maximum arg length, which can be emitted to log lines This should prevent from emitting too long lines when too long args are passed to logger.* functions. For example, too long MetricsQL queries or too long data samples.	2023-09-07 15:28:53 +02:00
dependabot[bot]	5a16d0936b	build(deps): bump actions/checkout from 3 to 4 (#4950 ) Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-09-07 15:23:52 +02:00
Github Actions	c12b24f2df	Automatic update operator docs from VictoriaMetrics/operator@9c461d2 (#4968 )	2023-09-07 13:04:49 +02:00
Aliaksandr Valialkin	1dd19df849	vendor: return back the latest version of golang.org/x/exp/slices, which works correctly with github.com/prometheus/prometheus/model/labels	2023-09-07 12:40:00 +02:00
Aliaksandr Valialkin	5cbb330555	vendor: run `make vendor-update`	2023-09-07 12:35:42 +02:00
Aliaksandr Valialkin	0bc0d2610c	go.mod: increase the minimum supported Go version from Go1.19 to Go1.20	2023-09-07 12:18:12 +02:00
Aliaksandr Valialkin	1dac069498	docs/CHANGELOG.md: return back accidentally deleted line at `45c0e4bb31`	2023-09-07 12:04:14 +02:00
Aliaksandr Valialkin	677612c6cb	all: update Go builder from Go1.21.0 to Go1.21.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved	2023-09-07 11:36:49 +02:00
Aliaksandr Valialkin	dddf092791	docs/CHANGELOG.md: clarify the scope of recent bugfixes	2023-09-07 11:25:18 +02:00
Aliaksandr Valialkin	8c656d0396	SECURITY.md: substitute v1.79.x LTS releases with v1.93.x LTS releases, because v1.79.x is deprecated	2023-09-07 11:19:40 +02:00
Aliaksandr Valialkin	fe739e771b	deployment/docker: properly build armv5 production builds for GOARCH=arm Pass GOARM=5 when building GOARCH=arm production builds, since the default value for this env var has been changed to GOARM=6 since Go1.21.0. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965 and https://github.com/golang/go/issues/62475	2023-09-07 11:19:40 +02:00
hagen1778	740725b7a3	docs: mention that `quantiles` can't be used in sharded mode https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `40c94b26dd`)	2023-09-07 10:59:19 +02:00
Haleygo	0212219f6c	vmalert: add `eval_offset` for group (#4693 ) Adds `eval_offset` attribute for Groups. If specified, Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]. The setting might be useful for cron-like rules which must be evaluated at specific moments of time. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409 Signed-off-by: Haley Wang <pipilong.25@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `45c0e4bb31`)	2023-09-07 10:59:14 +02:00
Aliaksandr Valialkin	e0923f988e	docs/CHANGELOG.md: document the bugfix at `7db72dd7e6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4947	2023-09-06 12:17:38 +02:00
Aliaksandr Valialkin	4d3c24492c	app/vmselect: run `make vmui-update`	2023-09-06 10:29:59 +02:00
Yury Molodov	6c96bebc23	vmui: fix render heatmap (#4957 )	2023-09-06 10:29:58 +02:00
Aliaksandr Valialkin	a0c4095ec9	docs/CaseStudies.md: update Grammarly case study with the newly published article https://www.grammarly.com/blog/engineering/monitoring-with-victoriametrics/ Follow-up `c0246b2e17`	2023-09-05 17:04:03 +02:00
Artem Navoiev	696f143eb5	use correct abbriviation for ESA legal doc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-09-05 16:59:47 +02:00
Artem Navoiev	40c795a1e3	change link to the enterprise legal doc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-09-05 16:58:51 +02:00
Github Actions	52859179b3	Automatic update operator docs from VictoriaMetrics/operator@b63f6e9 (#4946 )	2023-09-05 16:57:19 +02:00
hagen1778	9f4c4323de	docs: fix broken link in vmctl references Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-05 16:57:07 +02:00
Yury Molodov	30db332866	feat: add the option to see the latest queries (#4718 ) (#4759 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-05 16:56:53 +02:00
hagen1778	7e92821f94	docs: add `How Grammarly Improved Monitoring by Over 10x with VictoriaMetrics` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-09-05 16:56:14 +02:00
Aliaksandr Valialkin	dbcfffeedf	lib/flagutil: add Duration.Milliseconds() convenience function after `0c7d46d637` This function is a faster replacement for Duration.Duration().Milliseconds() call	2023-09-03 10:55:35 +02:00
Dima Lazerka	b97041c252	flagutil: Make .Msecs private (#4906 ) * Introduce flagutil.Duration To avoid conversion bugs * Fix tests * Clarify documentation re. month=31 days * Add fasttime.UnixTime() to obtain time.Time The goal is to refactor out the last usage of `.Msecs`. * Use fasttime for time.Now() * wip - Remove fasttime.UnixTime(), since it doesn't improve code readability and maintainability - Run `make docs-sync` for syncing changes from README.md to docs/ folder - Make lib/flagutil.Duration.Msec private - Rename msecsPerMonth const to msecsPer31Days in order to be consistent with retention31Days --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-03 10:37:57 +02:00
Github Actions	0b2d874eb7	Automatic update operator docs from VictoriaMetrics/operator@add9045 (#4941 )	2023-09-03 10:11:54 +02:00
Github Actions	e64ec149d6	Automatic update operator docs from VictoriaMetrics/operator@c32931b (#4940 )	2023-09-03 10:11:28 +02:00
Aliaksandr Valialkin	02f25ecf75	deployment: update VictoriaMetrics tag from v1.93.2 to v1.93.3 See https://docs.victoriametrics.com/CHANGELOG.html#v1933	2023-09-02 11:16:36 +02:00
Aliaksandr Valialkin	ed57b488d6	docs/CHANGELOG.md: document 1.93.3 release	2023-09-02 10:21:29 +02:00
Aliaksandr Valialkin	bd13b0f492	docs/CHANGELOG.md: document v1.87.8	2023-09-02 01:54:16 +02:00
Nikolay	fac272bc10	lib/vmselectapi: do not send empty label names for labelNames request (#4936 ) * lib/vmselectapi: do not send empty label names for labelNames request it breaks cluster communication, since vmselect incorrectly reads request buffer, leaving unread data on it https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4932 * typo fix * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-01 23:24:51 +02:00
Aliaksandr Valialkin	d7cd7cc998	docs/CHANGELOG.md: move the bugfix description for `7c19d01e9a` into proper place	2023-09-01 17:58:12 +02:00
Nikolay	7c19d01e9a	app/vminsert: properly close vmstorage connection (#4935 ) * app/vminsert: properly close vmstorage connection previously vmstorage may stuck in broken state until vminsert restarts since vmstorage was marked as read-only and connection was broken to it. checkReadonly function never marked connection as broken https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-01 17:56:41 +02:00
Aliaksandr Valialkin	ea254edb44	deployment: update VictoriaMetrics tag from v1.93.1 to v1.93.2 See https://docs.victoriametrics.com/CHANGELOG.html#v1932	2023-09-01 17:36:01 +02:00
Aliaksandr Valialkin	fcd7332b14	docs/CHANGELOG.md: document v1.93.2	2023-09-01 17:33:11 +02:00
Aliaksandr Valialkin	b9b2fbc7cd	app/vmselect: run `make vmui-update` after `c112dd7367`	2023-09-01 10:54:22 +02:00
Yury Molodov	9bd71b0f12	vmui: support for Prometheus data on the cardinality page (#4713 ) * feat: add cardinality support for prometheus (#4320) * docs/CHANGELOG.md: add cardinality support for prometheus --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-09-01 10:54:21 +02:00
Zakhar Bessarab	d8258be292	deployment/docker: add VictoriaLogs (#4929 ) * deployment/docker: add VictoriaLogs configuration Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker/victorialogs: remove outdated comment It was added in order to indicate that it is required to build VictoriaLogs manually before starting it at the time there was no public release available. Currently, there is a public tag and it is not required to build it from sources. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker/victorialogs/fluentbit: include log path in stream configuration Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker: add reference to monitoring setup for VictoriaLogs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-09-01 10:46:32 +02:00
Aliaksandr Valialkin	d8afd7fe98	Makefile: update golangci-lint from v1.51.2 to v1.54.2 See https://github.com/golangci/golangci-lint/releases/tag/v1.54.2	2023-09-01 10:25:49 +02:00
Dima Lazerka	1d60c236b1	Add flagutil.Duration to avoid conversion bugs (#4835 ) * Introduce flagutil.Duration To avoid conversion bugs * Fix tests * Comment why not .Seconds()	2023-09-01 09:30:30 +02:00
Aliaksandr Valialkin	953699d59a	app/vmauth: add tests for ResponseHeaders This is a follow-up for `b18eed3427` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825	2023-09-01 09:22:01 +02:00
Aliaksandr Valialkin	33e64d3207	app/vmctl: remove superflouos whitespace after `2853fac3f5`	2023-09-01 08:59:19 +02:00
Alexander Marshalov	9ba352da0f	vmauth: added ability to set and remove response headers (#4825 ) (#4914 ) * added ability to set and clear response headers (#4825) Signed-off-by: Alexander Marshalov <_@marshalov.org> * added ability to set and clear response headers (#4825) Signed-off-by: Alexander Marshalov <_@marshalov.org> * fix review comment Signed-off-by: Alexander Marshalov <_@marshalov.org> --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-09-01 08:57:52 +02:00
Dmytro Kozlov	986ad3d112	docs: follow up after `939952068b` (#4923 )	2023-09-01 08:52:45 +02:00
Aliaksandr Valialkin	d079780ec6	docs/VictoriaLogs/CHANGELOG.md: document `1c42154785` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762	2023-08-30 16:37:54 +02:00
Zakhar Bessarab	82060727ec	app/vlinsert/loki: add handler for healthcheck endpoint (#4885 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-08-30 16:37:54 +02:00
Nikolay	fbe2795670	app/vminsert: fixes readonly check (#4892 ) * app/vminsert: fixes readonly check previously vminsert doesn't check readOnly state for vmstorage, since check was never performed for nil buffer In this case every 30 second storage node loss readonly state and received some data. It caused re-routing and possible slow down for ingestion https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4870 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-30 16:24:24 +02:00
Nikolay	ae85b20c5b	lib/promscrape/k8s_sd: set resourceVersion to 0 by default for watch … (#4901 ) * lib/promscrape/k8s_sd: set resourceVersion to 0 by default for watch requests it must reduce load for kubernetes ETCD servers. Since requests without resourceVersion performs force cache sync at kubernetes API server with ETCD more info at https://kubernetes.io/docs/reference/using-api/api-concepts/\#semantics-for-watch https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4855 * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-30 16:04:14 +02:00
Github Actions	4eb42353bc	Automatic update operator docs from VictoriaMetrics/operator@18e080e (#4921 ) Co-authored-by: Alexander Marshalov <_@marshalov.org>	2023-08-30 15:21:05 +02:00
Aliaksandr Valialkin	1ca3b660f0	app/vmselect/promql: add support for `_` delimiters in numeric values For example, 1_234_567_890 is equivalent to 1234567890, while 1.234_567_890 is equivalent to 1.234567890	2023-08-30 14:35:58 +02:00
Github Actions	6b16b2545d	Automatic update operator docs from VictoriaMetrics/operator@d0c3ec7 (#4909 )	2023-08-30 14:15:27 +02:00
Aliaksandr Valialkin	3a2d035283	lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant Disallow parsing multitenant token at auth.NewToken(). Use auth.NewTokenPossibleMultitenant() at vminsert only. All the other callers should call auth.NewToken(), since they do not support multitenant token. This is a follow-up for `f0c06b428e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910	2023-08-30 14:13:51 +02:00
hagen1778	bda9699657	app/vmselect: follow-up after `f0c06b428e` Remove extra error message when auth token is nil. The default message about unsupported path should be more clear to the user who mistakenly requested /multitenant path. `f0c06b428e` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-29 21:52:50 +02:00
Zakhar Bessarab	f0c06b428e	app/vmselect: fix panic when using `/select/multitenant` endpoint (#4912 ) app/vmselect: fix panic when using `/select/multitenant` endpoint Such requests must be rejected as not found since vmselect does not support multitenant endpoint. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-08-29 21:48:12 +02:00
Nikolay	e711c8877e	deployment/docker: disable provenance in buildx (#4911 ) * deployment/docker: disable provenance in buildx it must fix an issue with multi-platform manifest generation at buildx >= 0.10 backward compatibility was broken and generated image cannot be used with docker systems that doesn't support oci. disabling attestat temporary fixes it. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907 https://docs.docker.com/build/attestations/slsa-provenance/ * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-29 16:32:13 +02:00
Aliaksandr Valialkin	4dcd2ff415	docs/VictoriaLogs/Roadmap.md: add alerting to TODO list	2023-08-29 13:26:32 +02:00
Aliaksandr Valialkin	503798da53	vendor: run `make vendor-update`	2023-08-29 13:16:20 +02:00
Aliaksandr Valialkin	b56294c174	lib/proxy: consistently use gopkg.in/yaml.v2 across all the code	2023-08-29 13:16:20 +02:00
Aliaksandr Valialkin	0609d8c8c5	Makefile: cleanup bin/ directory at the beginning of `make publish-release` command This is needed in order to prevent from non-build artifacts to be uploaded to Github release page	2023-08-29 13:16:20 +02:00
Aliaksandr Valialkin	5e8dfcf65e	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update` after recent changes to app/vmui	2023-08-29 12:58:58 +02:00
Aliaksandr Valialkin	339879edd0	lib/netutil/tcpdialer.go: reduce the code difference with enterprise branch	2023-08-29 12:45:57 +02:00
Aliaksandr Valialkin	19d61737c1	app/{vminsert,vmselect}: follow-up after `2b7b3293c1` - Document the change at docs/CHANGELOG.md - Set the default value for -vmstorageUserTimeout to 3 seconds. This is much better than the 0 value, which means that TCP connection to unreachable vmstorage could block for up to 16 minutes. - Document -vmstorageUserTimeout at docs/Cluster-VictoriaMetrics.md	2023-08-29 12:17:39 +02:00
Will Jordan	2b7b3293c1	Add `vmstorageUserTimeout` flags to configure TCP user timeout (Linux) (#4423 ) `TCP_USER_TIMEOUT` (since Linux 2.6.37) specifies the maximum amount of time that transmitted data may remain unacknowledged before TCP will forcibly close the connection and return `ETIMEDOUT` to the application. Setting a low TCP user timeout allows RPC connections quickly reroute around unavailable storage nodes during network interruptions.	2023-08-29 11:46:39 +02:00
Aliaksandr Valialkin	0aa0435d17	docs/VictoriaLogs/CHANGELOG.md: document the fix `8d50032dd6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4895	2023-08-29 11:29:11 +02:00
crossoverJie	64b27c9217	lib/logstorage: Set ptwHot to nil when the partition pointed by ptwHot is dropped (#4902 )	2023-08-29 11:22:53 +02:00
Roman Khavronenko	c81e90223c	dashboards: provide copies of Grafana dashboards alternated with Vict… (#4905 ) dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-29 11:20:16 +02:00
hagen1778	d70b346623	lib/promscrape: follow-up after `eabcfc9bcd` `-promscrape.cluster.membersCount` by default should be `1`, like every single vmagent is a cluster of one member on its own. The change additionally validates that user can't set `-promscrape.cluster.membersCount` to value lower than `1`. `eabcfc9bcd` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-29 11:18:12 +02:00
Haleygo	685c3d95e7	fix clusterMembersCount check (#4900 )	2023-08-29 11:15:50 +02:00
Aliaksandr Valialkin	1e1aa94ffb	lib/logstorage: eliminate data race when clearing s.ptwHot after deleting the corresponding partition The previous code could result in the following data race: 1. The s.ptwHot partition is marked to be deleted 2. ptw.decRef() is called on it 3. ptw.pt is set to nil 4. s.ptwHot.pt is accessed from concurrent goroutine, which leads to panic. The change clears s.ptwHot under s.partitionsLock in order to prevent from the data race. This is a follow-up for `8d50032dd6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4895	2023-08-29 11:12:07 +02:00
Aliaksandr Valialkin	c4b10109d7	docs/CHANGELOG.md: add links to stream parsing mode in descriptions for `6e8611f301` and `6788704152`	2023-08-29 10:47:40 +02:00
Aliaksandr Valialkin	66cb5930ae	docs/CHANGELOG.md: remove unneeded `utm_source` and `utm_medium` query args in the link to Google Lighthouse Remove the line about consistent rounding of values in vmui, since it looks like it has been broken and needs to be returned back. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4872#issuecomment-1696981947 for details. This is a follow-up for `e865989fa9`	2023-08-29 10:27:57 +02:00
Aliaksandr Valialkin	1159b31270	app/vmagent/remotewrite: do not retry request immediately on io.ErrUnexpectedEOF, since this error isn't returned on stale connection Also, mention the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139 in comments to the code in order to simplify further maintenance of this code. This is a follow-up for `992a1c0a3a`	2023-08-29 09:48:49 +02:00
Aliaksandr Valialkin	516627672e	docs/CHANGELOG.md: remove another blank line in order to reduce the difference with lts-1.93 branch	2023-08-29 09:48:48 +02:00
hagen1778	1464dda198	docs: fix typo for deduplication description in k8s guide Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a20234d2d0`)	2023-08-28 21:32:42 +02:00
Aliaksandr Valialkin	13ed621e36	docs/CHANGELOG.md: remove superflouos blank lines	2023-08-28 10:00:40 +02:00
Aliaksandr Valialkin	35cd8836c9	docs/CHANGELOG.md: move the bugfix line into correct place after `ddf87b32ed`	2023-08-28 10:00:40 +02:00
Aliaksandr Valialkin	48d5940f2d	docs/CHANGELOG.md: explicitly mention that the bug in 1.93.0 may lead to data loss Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873	2023-08-28 09:52:52 +02:00
Aliaksandr Valialkin	bba86475e8	docs/CHANGELOG.md: return back the line accidentally deleted at `6abd575cbe` The line has been originally added in `481a2c70fd`	2023-08-28 09:48:11 +02:00
Aliaksandr Valialkin	baea9da66b	docs/stream-aggregation.md: use `5m` instead of `300` in the example query for rate() calculation from "increase" results This makes the query easier to read and understand Follow-up for `0df506de54`	2023-08-28 09:37:34 +02:00
Aliaksandr Valialkin	453077808d	docs/vmauth.md: consistently prepend command-line flags with a single `-`	2023-08-28 09:28:24 +02:00
Daria Karavaieva	c730e1ed0c	vmanomaly guide docker compose run fix according licensing (#4876 ) * docker-compose run fix according to licensing * public v1.5.0 of vmanomaly * new file list image * 1.93.1 version of VM	2023-08-28 09:23:54 +02:00
Github Actions	0ba4c1c65c	Automatic update operator docs from VictoriaMetrics/operator@d40882c (#4894 ) Co-authored-by: Alexander Marshalov <_@marshalov.org>	2023-08-28 09:17:37 +02:00
Aliaksandr Valialkin	1e34318dca	docs/CHANGELOG.md: clarify the description of `b7d07e5acf` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4817	2023-08-28 09:12:49 +02:00
crossoverJie	db0ae3fffb	lib/logstorage: add nil check for ptwHot.pt (#4896 ) (cherry picked from commit `cde5029bce`)	2023-08-27 09:14:28 +02:00
Github Actions	40b1f9611c	Automatic update operator docs from VictoriaMetrics/operator@e00fe21 (#4891 ) (cherry picked from commit `ef6468584c`)	2023-08-27 09:06:02 +02:00
Zakhar Bessarab	46e86add2f	lib/promscrape/client: sync timeout for HostClient and http.Client (#4889 ) Initially, stream parse mode was reading data from response and parsing it on flight. This was causing longer delay to read the whole response and required increasing timeout value to allow data processing while reading. So that `908e35affd` increased timeout value to fix this. But after `74c00a8762` response in stream parse mode is saved into memory and then parsed eliminating necessity of having timeout value higher that for usual scrape. Updates: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4847 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `6e8611f301`)	2023-08-27 09:06:00 +02:00
hagen1778	dc837cc65f	docs: follow-up after `72167a697e` `72167a697e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e865989fa9`)	2023-08-27 09:05:58 +02:00
Yury Molodov	2aff0ce327	vmui: ui improvements (#4872 ) * vmui: chart refactoring to enhance code structure * vmui: improve ui (cherry picked from commit `72167a697e`)	2023-08-27 09:05:55 +02:00
hagen1778	6c99e5445d	docs: make docs-sync Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `65415b56af`)	2023-08-27 09:05:51 +02:00
hagen1778	ff20d3e97c	docs: update remote-read api docs Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `f4577005be`)	2023-08-27 09:05:51 +02:00
Dmytro Kozlov	939952068b	docs: add migration guide from Promscale (#4865 ) Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com (cherry picked from commit `0b0e0bb50e`)	2023-08-27 09:05:44 +02:00
hagen1778	b42187cf82	docs: typo fix for vmauth Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ffbebfdfe6`)	2023-08-27 09:05:41 +02:00
hagen1778	49327b584d	docs: mention that vmauth doesn't follow redirects See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4868 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4d316a23ae`)	2023-08-27 09:05:37 +02:00
hagen1778	b18e9b5bb0	app/vmagent: follow-up after `6788704152` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4884 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4ebe8bb1d5`)	2023-08-27 09:05:22 +02:00
Zakhar Bessarab	1242460fa6	lib/promscrape/client: make User-Agent consistent between fasthttp and native client (#4886 ) User agent was not set for native client which resulted in using one provided by Golang. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4884 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `6788704152`)	2023-08-27 09:05:08 +02:00
hagen1778	33bf28e1bd	app/vmagent: fix comment typo after `992a1c0a3a` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `757ae4275b`)	2023-08-27 09:05:04 +02:00
Roman Khavronenko	b9a2512ac3	vmagent: retry failed write request on the closed connection (#4857 ) * vmagent: retry failed write request on the closed connection Retry failed write request on the closed connection immediately, without waiting for backoff. This should improve data delivery speed and reduce amount of error logs emitted by vmagent when using idle connections. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: retry failed write request on the closed connection Re-instantinate request before retry as body could have been already spoiled. Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com> (cherry picked from commit `992a1c0a3a`)	2023-08-27 09:04:59 +02:00
Roman Khavronenko	68150655d2	vmalert: correctly re-instantinate HTTP req on retries (#4864 ) * vmalert: correctly re-instantinate HTTP req on retries Previosly, request retry to datasource re-used existing HTTP request. But if request object was already partially processed (body was read), then retry will be unsuccessful. The change re-instantinates HTTP request object before retry. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: review fix Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ddf87b32ed`)	2023-08-27 09:04:42 +02:00
Github Actions	74967aa704	Automatic update operator docs from VictoriaMetrics/operator@8a25794 (#4883 ) (cherry picked from commit `2122eb18fe`)	2023-08-27 09:04:34 +02:00
Aliaksandr Valialkin	c813b5e4b1	lib/promscrape: add -promscrape.cluster.memberLabel command-line flag This flag allows specifying an additional label to add to all the scraped metrics. The flag must contain label name to add. The label value will be equal to -promscrape.cluster.memberNum. This functionality can help when there is a need to differentiate metrics scraped by distinct vmagent instances in the cluster according to https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247#issuecomment-1692279393	2023-08-24 22:04:34 +02:00
Aliaksandr Valialkin	18f8e90bd8	docs/stream-aggregation.md: typo fix after `54f522ac25`	2023-08-24 22:04:24 +02:00
hagen1778	30bc1752dd	docs: mention 1.93.0 contains a bug Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `59dee2e714`)	2023-08-23 16:44:23 +02:00
hagen1778	e49c92fd1f	marketplace: bump DO to 1.93.1 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `df6d27650e`)	2023-08-23 16:44:23 +02:00
Nikolay	50584cfeff	docs: release docs 1.93.1 (#4875 ) * docs: mention v1.93.1 release * deployment/docker: bumps image for v1.93.1 release (cherry picked from commit `6abd575cbe`)	2023-08-23 16:44:23 +02:00
hagen1778	4cf8e58e08	docs: mention breaking change to indexdb intorduced in 1.92.0 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `946e370b26`)	2023-08-23 16:44:22 +02:00
Nikolay	1a943bb16a	lib/storage: properly caclucate nextRotationTimestamp (#4874 ) cause of typo unix millis was used instead of unix for current timestamp calculation https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873 (cherry picked from commit `c5aac34b68`)	2023-08-23 13:29:32 +02:00
hagen1778	92f158a2f5	docs: mention `increase` as alternative to `rate` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `0df506de54`)	2023-08-23 13:29:32 +02:00
Zakhar Bessarab	0838607897	docs/vmanomaly: update licensing flags description Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `7961479900`)	2023-08-23 13:29:32 +02:00
Yury Molodov	c527d2a549	vmui: change warning display for text fields (#4848 ) (#4863 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4848 (cherry picked from commit `ca44b8da1f`)	2023-08-21 15:50:20 +02:00
Yury Molodov	931c63f602	vmui: chart refactoring to enhance code structure (#4830 ) (cherry picked from commit `8287749c05`)	2023-08-21 15:50:19 +02:00
hagen1778	f48962e834	vmselect: follow-up after `7349f18c55` `7349f18c55` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `ea2fbcf0e6`)	2023-08-21 15:50:19 +02:00
Lapo Luchini	44ce25565a	Fix typo in `vmauth` docs. (cherry picked from commit `297ef605ef`)	2023-08-21 15:50:19 +02:00
Zakhar Bessarab	eb9bc2969d	docs/vmanomaly: clarify offline verification flag description Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `9169f65521`)	2023-08-21 15:50:18 +02:00
Zakhar Bessarab	ca2600627e	docs: add vmanomaly docs about monitoring - add monitoring section - add reference to monitoring section from licensing monitoring Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `8ea4ae7dbb`)	2023-08-21 15:50:18 +02:00
Zakhar Bessarab	23f443d5cd	docs: add vmanomaly docs about licensing Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `ee747f4f42`)	2023-08-21 15:50:18 +02:00
Tamara Vashchuk	6a59737e96	vmui: Add button to prettify query (#4694 ) * Add button to prettify query Just capitalizes query text for now * Add /prettify-query API handler * Replace UI pretiffier using prettifier API * Add showing server errors Had to pass setQueryErrors from useFetchQuery.ts * Use serverUrl from global AppState * Change icon to AutoAwsome icon + added style change color when button is active * Add sync/await to prettifyQuery function * Doc public function for lint * Minor async fix * Removed extra blank lines * Extract usePrettifyQuery hook * Made more generic style for :active button * Refactor usePrettifyQuery However, prettify errors don't clean up query errors, but should * Add prettyQuery functionality to CHANGELOG.md * Reuse queryErrors * Unhide errors on start --------- Co-authored-by: Tamara <toma.vashchuk@gmail.com> (cherry picked from commit `7349f18c55`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-21 15:50:17 +02:00
Github Actions	ad91666858	Automatic update operator docs from VictoriaMetrics/operator@76d5956 (#4854 ) (cherry picked from commit `7d8b6fbe20`)	2023-08-18 13:39:43 +02:00
hagen1778	c4d22dfefb	docs: exclude assets/README.md exclude assets/README.md from publishing on the docs website as its purpose is different to other docs. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `e9d246f367`)	2023-08-18 13:38:55 +02:00
Dmytro Kozlov	1929d3bca9	lib/protoparser: handle unexpected EOF error when parsing lines in prometheus exposition format (#4851 ) Previously only io.EOF was handled, and io.ErrUnexpectedEOF was ignored, but it may happen if the client interrupts the connection. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4817 (cherry picked from commit `b7d07e5acf`)	2023-08-18 08:56:27 +02:00
Aliaksandr Valialkin	07a3030856	docs/stream-aggregation.md: clarify the usage of `-remoteWrite.label` after the fix at `a27c2f3773` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247	2023-08-17 15:19:04 +02:00
Aliaksandr Valialkin	50820ab1aa	app/vmagent/remotewrite: follow-up after `a27c2f3773` - Fix Prometheus-compatible naming after applying the relabeling if -usePromCompatibleNaming command-line flag is set. This should prevent from possible Prometheus-incompatible metric names and label names generated by the relabeling. - Do not return anything from relabelCtx.appendExtraLabels() function, since it cannot change the number of time series passed to it. Append labels for the passed time series in-place. - Remove promrelabel.FinalizeLabels() call after adding extra labels to time series, since this call has been already made at relabelCtx.applyRelabeling(). It is user's responsibility if he passes labels with double underscore prefixes to -remoteWrite.label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247	2023-08-17 14:44:35 +02:00
Aliaksandr Valialkin	67dd975be5	lib/envflag: do not allow unsupported form for boolean command-line flags in the form `-boolFlag value` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4845	2023-08-17 13:37:05 +02:00
Alexander Marshalov	73287a7c3a	vmagent: fixed premature release of the context (after #4247 / #4824 ) (#4849 ) Follow-up after `a27c2f3773` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247 Signed-off-by: Alexander Marshalov <_@marshalov.org> (cherry picked from commit `1e1a30ed7f`)	2023-08-17 12:16:04 +02:00
Dmytro Kozlov	15bc40216d	app/vmctl: fix migration process if tenant have no data (#4799 ) app/vmctl: don't interrupt migration process if tenant has no data Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Alexander Marshalov <_@marshalov.org> (cherry picked from commit `39623ae428`)	2023-08-16 14:56:52 +02:00
Roman Khavronenko	505ef72e69	vmbackup: correctly check if specified `-dst` belongs to specified `-storageDataPath` (#4841 ) See this issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `6da32a27ac`)	2023-08-16 14:56:52 +02:00
Alexander Marshalov	58cf862b05	fixed applying `remoteWrite.label` for pushed metrics (#4247 ) (#4824 ) vmagent: properly add extra labels before sending data to remote storage labels from `remoteWrite.label` are now added to sent metrics just before they are pushed to `remoteWrite.url` after all relabelings, including stream aggregation relabelings (#4247) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247 Signed-off-by: Alexander Marshalov <_@marshalov.org> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `a27c2f3773`)	2023-08-15 13:48:19 +02:00
Github Actions	80ec24a088	Automatic update operator docs from VictoriaMetrics/operator@9f41076 (#4840 ) (cherry picked from commit `c884311cf5`)	2023-08-15 13:48:18 +02:00
Arseny	1ed599e171	docs/Articles.md: add a link to https://rtfm.co.ua/en/victoriametrics-deploying-a-kubernetes-monitoring-stack/ (#4836 ) (cherry picked from commit `93109842c6`)	2023-08-15 09:21:31 +02:00
hagen1778	ae92d46f3c	dashboard: fix display of ingested rows rate Fix display of ingested rows rate for `Samples ingested/s` and `Samples rate` panels for vmagent's dasbhoard. Previously, not all ingested protocols were accounted in these panels. An extra panel `Rows rate` was added to `Ingestion` section to display the split for rows ingested rate by protocol. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `481a2c70fd`)	2023-08-15 09:21:30 +02:00
Artem Navoiev	8ed8dba1b3	fix docs ordering Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `507879380b`)	2023-08-15 09:21:30 +02:00
Aliaksandr Valialkin	4b1f01e45d	lib/promrelabel: properly replace `:` char with `_` in metric names when -usePromCompatibleNaming command-line flag is set This addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113#issuecomment-1275077071 comment from @johnseekins	2023-08-14 16:18:17 +02:00
Aliaksandr Valialkin	e8fe00d39e	lib/promrelabel: stop emitting DEBUG log lines when parsing `if` expressions These lines were accidentally left in the commit `62651570bb` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635	2023-08-14 16:18:17 +02:00
Aliaksandr Valialkin	bde876f7c9	all: refer to https://docs.victoriametrics.com/#resource-usage-limits in the error message about -search.max* limit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4827	2023-08-14 02:02:12 -07:00
Aliaksandr Valialkin	1361239393	app/vmbackup: add ability to make server-side copying of existing backups	2023-08-13 17:26:26 -07:00
Aliaksandr Valialkin	ad65c769da	README.md: sync with docs/Cluster-VictoriaMetrics.md	2023-08-12 16:26:41 -07:00
Aliaksandr Valialkin	8ce9cea087	docs/Articles.md: add a link to https://www.forbes.com/sites/adrianbridgwater/2023/07/05/the-agility-in-cloud-observability/	2023-08-12 16:25:33 -07:00
Aliaksandr Valialkin	ea1a194c46	docs/Articles.md: add a link to https://rtfm.co.ua/en/victoriametrics-an-overview-and-its-use-instead-of-prometheus/	2023-08-12 16:25:33 -07:00
Aliaksandr Valialkin	28e7a4024d	docs/CHANGELOG.md: document that v1.93.x is a new line of LTS releases	2023-08-12 15:32:03 -07:00
Aliaksandr Valialkin	d2c1515e86	docs/CHANGELOG.md: document changes in the v1.87.7 LTS release	2023-08-12 14:49:28 -07:00
Aliaksandr Valialkin	477db57a05	docs/CHANGELOG.md: document LTS release v1.79.14 See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.14	2023-08-12 12:28:32 -07:00
Aliaksandr Valialkin	8fccdcfed5	deployment: update Docker image tags for VictoriaMetrics components from v1.92.1 to v1.93.0 See https://docs.victoriametrics.com/CHANGELOG.html#v1930	2023-08-12 12:26:12 -07:00
Aliaksandr Valialkin	f7f29a1840	app/vmctl: document that -vm-native-step-interval command-line option now supports `week` value This is a follow-up for `d322ee4b35` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738	2023-08-12 07:33:13 -07:00
Aliaksandr Valialkin	9ac0d980f9	docs/CHANGELOG.md: cut v1.93.0	2023-08-12 06:01:17 -07:00
Aliaksandr Valialkin	181a6152ae	deployment/docker/Makefile: upgrade base Docker image from alpine:3.18.2 to alpine:3.18.3 See https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html	2023-08-12 06:00:07 -07:00
Aliaksandr Valialkin	b437f96c5a	docs/CHANGELOG.md: split changelog into per-year pages in order to keep the size of CHANGELOG pages under control Make sure that links to particular releases - https://docs.victoriametrics.com/CHANGELOG.html#vXXYY - continue working.	2023-08-12 05:49:51 -07:00
Nikolay	bb2885d57d	lib/protoparser/openetelemetry: fixes panic (#4821 ) Opentelemetry format allows histograms with non-counter buckets. In this case it makes no sense to add buckets into database and save only counter with _count suffix. It could be used as gauge. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4814 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-12 05:10:05 -07:00
Nikolay	476286385f	opentelemetry: return human readable error for json encoding. (#4822 ) Opentelemetry parser supports only protobuf atm. Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-12 05:06:19 -07:00
Nikolay	89fcb7baf0	lib/promscrape: adds validation for proxy_url scheme (#4823 ) * lib/promscrape: adds validation for proxy_url scheme adds tests https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4811 * Update lib/proxy/proxy.go * Update lib/proxy/proxy.go --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-12 05:05:56 -07:00
Aliaksandr Valialkin	5c80b11c15	app/vmselect: prevent from panic when lookbehind window inside rollup function is parsed into negative value Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4795	2023-08-12 04:49:56 -07:00
Aliaksandr Valialkin	0ee8a9120a	lib/flagutil: add defaultValue arg to NewArray{Int,Bytes,Duration} functions The defaultValue is printed in the flag description when passing -help to the app. This is a follow-up for `aef31f201a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4776	2023-08-12 04:19:34 -07:00
Aliaksandr Valialkin	6092b98849	vendor: run `make vendor-update`	2023-08-11 08:38:36 -07:00
Aliaksandr Valialkin	37af7d4ed3	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update` after `86f1459ca6`	2023-08-11 07:01:15 -07:00
Zakhar Bessarab	bae701e612	app/vlinsert/elasticsearch: add a command-line flag to provide ES version (#4778 ) * app/vlinsert/elasticsearch: add a command-line flag to provide ES version Adds a flag which will allow to change version which will be reported by ES endpoint for compatibility checks performed by external logs shippers(such as filebeat). See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Document the -elasticsearch.version command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-11 06:57:01 -07:00
Zakhar Bessarab	15b1810dc8	lib/promrelabel: fix relabeling if clause (#4816 ) * lib/promrelabel: fix relabeling if clause being applied to labels outside of current context Relabeling is applied to each metric row separately, but in order to lower amount of memory allocations it is reusing labels. Functions which are working on current metric row labels are supposed to use only current metric labels by using provided offset, but if clause matcher was using the whole labels set instead of local metrics. This leaded to invalid relabeling results such as one described here: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/CHANGELOG.md: document the bugfix Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-08-11 06:44:46 -07:00
Aliaksandr Valialkin	4470856cc4	all: update Go builder from Go1.20.7 to Go1.21.0 See https://tip.golang.org/doc/go1.21 and https://go.dev/blog/go1.21	2023-08-11 06:26:27 -07:00
Aliaksandr Valialkin	02a54dbe63	app/vmagent/remotewrite: go fmt	2023-08-11 06:26:12 -07:00
Aliaksandr Valialkin	b9e34a1386	docs/CHANGELOG.md: add a link to stream aggregation for the description of the bugfix at `a4a1884237` This makes the description more clear. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4804	2023-08-11 05:48:10 -07:00
Aliaksandr Valialkin	f639c01435	docs/CHANGELOG.md: add missing context to the description of the fix at `be5c4818f5`	2023-08-11 05:47:49 -07:00
Aliaksandr Valialkin	fa400f83b6	lib/httpserver: properly quote the returned address from GetQuotedRemoteAddr() for requests with X-Forwarded-For header Make sure that the quoted address can be used as JSON string. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676#issuecomment-1663203424 This is a follow up for `252643d100` and `ac0b7e0421` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676	2023-08-11 05:47:28 -07:00
Aliaksandr Valialkin	d02fb47c2d	app/vmagent/remotewrite: keep in sync the default value for -remoteWrite.sendTimeout option in the description with the actually used timeout This is a follow-up for `aef31f201a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4776	2023-08-11 05:46:27 -07:00
Aliaksandr Valialkin	ca49f23a45	docs/Release-Guide.md: document that changes must be synced between branches immediately after the commit in any branch	2023-08-11 05:46:04 -07:00
Aliaksandr Valialkin	e9251bbe45	docs/CaseStudies.md: update Naver case study after `0c3d61b211` - Fix a link to case study at the top of docs/CaseStudies.md - Remove non-essential text - Add Naver video and slides about VictoriaMetrics to docs/Articles.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4755	2023-08-11 05:45:43 -07:00
Aliaksandr Valialkin	2601930f20	app/vmctl: follow-up after `5aed369132` - Fix default value for --remote-read-disable-path-append - Clarify description for the change at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655 TODO: address the comment at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4744	2023-08-11 05:45:22 -07:00
Aliaksandr Valialkin	d0e5976966	docs/CHANGELOG.md: remove superflouos information from the line, which describes the upgrade from Go1.20.6 to Go1.20.7	2023-08-11 05:44:24 -07:00
Aliaksandr Valialkin	e0422365c0	docs/CHANGELOG.md: clarify the change at `e3ef3df938` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697	2023-08-11 05:43:58 -07:00
Zakhar Bessarab	9efc077214	vmbackupmanager: fixes for windows compatibility (#641 ) * app/vmbackupmanager/storage: fix path join for windows See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/backup: fixes for windows support - close dir before running os.RemoveAll. Windows FS does not allow to delete directory before all handles will be closed. - add path "normalization" for local FS to use the same format of paths for both *unix and Windows See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-08-11 05:43:28 -07:00
Github Actions	bc2145da23	Automatic update operator docs from VictoriaMetrics/operator@9ea59e2 (#4813 ) Co-authored-by: Alexander Marshalov <_@marshalov.org>	2023-08-11 05:42:51 -07:00
Github Actions	c9dc6e2bbd	Automatic update operator docs from VictoriaMetrics/operator@2a03bde (#4812 )	2023-08-11 05:41:39 -07:00
hagen1778	f847e2f91d	docs: mention `honor_timestamps` change in changelog https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 05:41:10 -07:00
Zakhar Bessarab	bffec2fc02	{vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency (#4809 ) {vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency Sync behavior for dropInput and keepInput flags between single-node and vmagent. Fix vmagent not respecting dropInput flag and reverse logic for keepInput.	2023-08-11 05:40:06 -07:00
Yury Molodov	088d08ac4a	vmui: hide "Logs Explorer" for the base build (#4761 ) LogsExplorer should be a part of VictoriaLogs binaries, as well as VMUI is now part of VictoriaMetrics binaries.	2023-08-11 05:00:08 -07:00
Yury Molodov	cf0077b552	vmui: allow displaying the full error message on click (#4760 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4719	2023-08-11 04:59:34 -07:00
Github Actions	336744a93e	Automatic update operator docs from VictoriaMetrics/operator@ca27728 (#4802 )	2023-08-11 04:58:55 -07:00
Haleygo	404c330c5a	docs: add changelog for `4c815ed59b` (#4805 )	2023-08-11 04:58:27 -07:00
Abirdcfly	70780306b3	vmalert: fix `vmalert_remotewrite_send_duration_seconds_total` metric value (#4801 ) The deferred call's arguments are evaluated immediately, but the function call is not executed until the surrounding function returns. Signed-off-by: Abirdcfly <fp544037857@gmail.com>	2023-08-11 04:58:00 -07:00
hagen1778	1786a703fd	vmalert: mention `vmalert_iteration_duration_seconds` metric in README Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:57:24 -07:00
Haleygo	74d5622606	vmalert: fix redundant clean up move (#4803 ) Follow-up after `55ae2c2d57`	2023-08-11 04:56:57 -07:00
Roman Khavronenko	4c91773a15	vmalert: cleanup config reload metrics handling (#4790 ) * rename `configErr` to `lastConfigErr` to reduce confusion * add tests to verify metrics and msg are set properly * fix mistake when config success metric wasn't restored after an error Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:56:21 -07:00
Haleygo	c0fdd73313	vmalert: fix uncleaned tmp files in tests (#4788 )	2023-08-11 04:55:42 -07:00
Github Actions	a204717f1d	Automatic update operator docs from VictoriaMetrics/operator@7e12a16 (#4783 )	2023-08-11 04:55:10 -07:00
Zakhar Bessarab	0747ca0595	docs: make phrase about dedup and evaluation interval relation less obscure (#4781 ) Value of `-dedup.minScrapeInterval` comand-line flag must be higher than `evaluation_interval` in order to make sure that only one sample on each evaluation will be left after deduplication. Moreover, value of `-dedup.minScrapeInterval` must be a multiple of vmalert's `evaluation_interval` in order to make sure that samples will be aligned between deduplication window periods. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4774#issuecomment-1663940811 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-08-11 04:54:43 -07:00
hagen1778	05b4fbf0b5	dashboards: correctly calculate `Bytes per point` value Correctly calculate `Bytes per point` value for single-server and cluster VM dashboards. Before, the calculation mistakenly accounted for the number of entries in indexdb in denominator, which could have shown lower values than expected. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:53:56 -07:00
Alexander Marshalov	d90dae2a68	add info about `remoteWrite.sendTimeout` default value (#4776 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-08-11 04:53:16 -07:00
Karan Sharma	e40231daad	fix: remove healthcheck in vector docker-compose (#4772 )	2023-08-11 04:45:59 -07:00
Roman Khavronenko	7220f70506	security: bump go version from 1.20.6 to 1.20.7 (#4773 ) The update includes a security fix to the crypto/tls package, as well as bug fixes to the assembler and the compiler. See the list of issues addressed in Go1.20.7 here: https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:44:01 -07:00
hagen1778	a7f0b8436c	dashboards: add panels for absoulte value of mem and cpu usage by vmalert See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4627 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:43:01 -07:00
hagen1778	15727b3d03	deployment: bump components version to 1.92.1 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:41:44 -07:00
hagen1778	f2b06484f2	alerts: move `ConcurrentFlushesHitTheLimit` alert to health alerts The `ConcurrentFlushesHitTheLimit` could be related to components like vminsert, vmstorage, vm-single-node and vmagent. Moving this alert to the `health` section of alerts will be benefitial for all components and will remove the duplicates from single/cluster alerts. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:39:28 -07:00
hagen1778	be7001351c	alerts: add docs section for the full list of alerting rules The change also includes update of all references in other docs to the alerting rules. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:38:30 -07:00
hagen1778	2f05be37b3	dashboards: add `Concurrent inserts` panel to vmagent's dasbhoard The new panel supposed to show whether the number of concurrent inserts processed by vmagent isn't reaching the limit. The panel contains recommendation what to do if limit is reached. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:36:40 -07:00
Github Actions	9486affadf	Automatic update operator docs from VictoriaMetrics/operator@29a7495 (#4770 )	2023-08-11 04:35:36 -07:00
Jan Kielmann	24515715c4	Fix spelling mistake in opentelemetry ingestion API path	2023-08-11 04:34:41 -07:00
Artem Navoiev	de33cb36d6	add link to the slides for Naver case study Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-08-11 04:33:16 -07:00
hagen1778	d06c224d2e	docs: follow-up after `df37a47d4b` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4415 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 04:32:36 -07:00
SunKyu Lee	1d2f0e8c7b	vmauth: add broken backend backoff flag (#4416 ) vmauth: allow configuring deadline for a backend to be excluded from the rotation The new flag `-failTimeout` allows overriding default time for a bad backend to be excluded from rotation. The override option could be useful for systems where it is expected for backends to be off for significant periods of time. Co-authored-by: Zakhar Bessarab <zekker6@gmail.com>	2023-08-11 04:30:23 -07:00
Yury Molodov	d443ab5ae1	vmui: display partial response warning (#4742 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4721	2023-08-11 04:17:27 -07:00
Artem Navoiev	3a8842cdcc	docs: add Naver Case Study (#4755 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-08-11 04:13:38 -07:00
Anton Tykhyy	2107b3133e	Remove some repetitions in docker Makefile (#4764 )	2023-08-11 04:00:26 -07:00
Dmytro Kozlov	dabce3cc5d	app/vmctl: add flag where use can define path to the source remote read protocol (#4744 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655	2023-08-11 03:56:34 -07:00
Roman Khavronenko	5a5785292c	vmctl: allow disabling binary export protocol (#4716 ) Binary export API protocol can be disabled via `-vm-native-disable-binary-protocol` cmd-line flag when migrating data from VictoriaMetrics. Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need to be configured at `-vm-native-src-addr` side and `-vm-native-disable-binary-protocol` should be set on vmctl side. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-08-11 03:19:20 -07:00
Zakhar Bessarab	3099114c7c	docs: add changelog entry for #4704 (#4753 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-08-11 03:17:02 -07:00
zhaojinxin409	aca4f69023	Update stream-aggregation.md for `speed` misspelling (#4752 )	2023-08-11 03:16:30 -07:00
Github Actions	9d5f77cca0	Automatic update operator docs from VictoriaMetrics/operator@dd980c8 (#4751 )	2023-08-11 03:13:14 -07:00
Nikolay	b0977b07fb	app/vminsert: adds note for dropSamplesOnOverload flag (#4797 ) Adds note for dropSamplesOnOverload flag that are samples dropped before replication	2023-08-10 12:18:29 +02:00
Dmytro Kozlov	8d0576c714	app/vmctl: add support the `week` step for time-based chunks (#4743 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738	2023-07-31 08:00:11 -07:00
Dmytro Kozlov	2d507a8757	app/vmctl: remove ping from remote read protocol (#4749 ) Ping method was never used, so we remove it.	2023-07-31 07:59:03 -07:00
Roman Khavronenko	02dba5f728	vmalert: remove deprecated in v1.79.0 web links with `*/status` suffix (#4747 ) Links of form `/api/v1/<groupID>/<alertID>/status` were deprecated in favour of `/api/v1/alerts?group_id=<>&alert_id=<>` links in v1.79.0. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825 This change removes code responsible for deprecated functionality. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:58:37 -07:00
hagen1778	c6acb5b6bc	docs: rm typo of naming vmalert as a stateless service Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:57:42 -07:00
Aliaksandr Valialkin	8b4bf5d269	app/vlstorage: expose vl_data_size_bytes metric at /metrics page for tracking the on-disk data size (both indexdb and the data itself)	2023-07-31 07:56:16 -07:00
Roman Khavronenko	85eb62a2ec	vmalert: remove deprecated in v1.61.0 `-rule.configCheckInterval` (#4745 ) Use `-configCheckInterval` command-line flag instead. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:55:54 -07:00
hagen1778	9d4c838412	vendor: `make vendor-update` Follow-up after `3f6efab6ae` Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4441 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:55:01 -07:00
hagen1778	e716ae1921	docs: mention `3f6efab6ae` in changelog Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:53:32 -07:00
Damon07	4c509c0b89	{app/vmselect,docs}: support share_eq_over_time#4441 (#4725 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4441 Co-authored-by: wangm <wangmm@tuya.com>	2023-07-31 07:51:09 -07:00
hagen1778	ee13ee1658	docs: add link to release-guide follow-up Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:48:13 -07:00
hagen1778	7a9c211b36	docs: remove anchors from the 1.92 release Adding anchors to the 1.92 changelog breaks consistency of navigation section at https://docs.victoriametrics.com/CHANGELOG.html All other releases do not have subsections, so should 1.92. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-31 07:47:22 -07:00
Aliaksandr Valialkin	693866a516	docs/CHANGELOG.md: move bugfix description to `tip` chapter, since it isnt released yet Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697	2023-07-28 23:01:51 -07:00
Aliaksandr Valialkin	3e62c71e8c	lib/promscrape: add a comment why `honor_timestamps` is set to false by default This should prevent from returning it back to true in the future Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697	2023-07-28 21:36:55 -07:00
Aliaksandr Valialkin	ee98f9ae66	lib/promscrape: use local scrape timestamp for scraped metrics unless `honor_timestamps: true` is set explicitly This fixes the case with gaps for metrics collected from cadvisor, which exports invalid timestamps, which break staleness detection at VictoriaMetrics side. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697 , https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1654614799 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697#issuecomment-1656540535 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1773	2023-07-28 21:11:46 -07:00
Aliaksandr Valialkin	89ccf19b70	lib/storage: update nextRotationTimestamp relative to the timestamp of the indexdb rotation Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563	2023-07-28 19:48:42 -07:00
Aliaksandr Valialkin	dfe4816e8c	docs/VictoriaLogs/data-ingestion/README.md: add checkboxes for Loki format support across log shippers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4728	2023-07-28 11:31:34 -07:00
Aliaksandr Valialkin	feb879eb99	docs/VictoriaLogs: add missing information about Promtail support for data ingestion Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4728	2023-07-28 11:17:14 -07:00
hagen1778	418042cae2	docs: fix step references in Release-Guide.md Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-28 11:16:59 -07:00
Aliaksandr Valialkin	7b01e28371	app/vmalert: use proper timestamp in setConfigSuccess()	2023-07-28 11:16:14 -07:00
Zakhar Bessarab	c5c539f973	docs/CHANGELOG.md: cut v1.92.1 (#4735 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `8f257889cc`)	2023-07-28 11:09:56 +02:00
Roman Khavronenko	303d3616ec	vmalert: revert unittest feature (#4734 ) * Revert "vmalert: unittest support stale datapoint (#4696)" This reverts commit `0b44df7ec8`. * Revert "docs: specify min version and limitations for vmalert's unit tests" This reverts commit `a24541bd` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Revert "vmalert: init unit test (#4596)" This reverts commit `da60a68d` Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: mention unittest revert in changelog Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `9f1b9b86cc`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-28 11:09:52 +02:00
Aliaksandr Valialkin	63d9a92d3d	docs/vmalert.md: fix broken links to `Web` chapter	2023-07-27 18:05:25 -07:00
Aliaksandr Valialkin	174f6793f3	docs/CHANGELOG.md: delimit changes from update notes	2023-07-27 17:06:37 -07:00
Aliaksandr Valialkin	f850421103	deployment: update VictoriaMetrics release tag from v1.91.3 to v1.92.0	2023-07-27 14:58:57 -07:00
Aliaksandr Valialkin	fd4b10330a	docs/CHANGELOG.md: cut v1.92.0	2023-07-27 14:55:52 -07:00
Aliaksandr Valialkin	1f30f53df2	lib/promscrape/discovery: close unused HTTP connections to service discovery servers This should prevent from connection leaks See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724	2023-07-27 14:47:55 -07:00
Nikolay	85de94e85c	lib/protoparser: adds opentelemetry parser (#2570 ) * lib/protoparser: adds opentelemetry parser app/{vmagent,vminsert}: adds opentelemetry ingestion path Adds ability to ingest data with opentelemetry protocol protobuf and json encoding is supported data converted into prometheus protobuf timeseries each data type has own converter and it may produce multiple timeseries from single datapoint (for summary and histogram). only cumulative aggregationFamily is supported for sum(prometheus counter) and histogram. Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> updates deps fixes tests wip wip wip wip lib/protoparser/opentelemetry: moves to vtprotobuf generator go mod vendor lib/protoparse/opentelemetry: reduce memory allocations * wip - Remove support for JSON parsing, since it is too fragile and is rarely used in practice. The most clients send OpenTelemetry metrics in protobuf. The JSON parser can be added in the future if needed. - Remove unused code from lib/protoparser/opentelemetry/pb and lib/protoparser/opentelemetry/proto - Do not re-use protobuf message between ParseStream() calls, since there is high chance of high fragmentation of the re-used message because of too complex nested structure of the message. * wip * wip * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-27 13:37:15 -07:00
Aliaksandr Valialkin	ce18e9b2c4	app/vmalert: make golangci-lint happy after `ae0e4a8c90`	2023-07-27 13:27:36 -07:00
Aliaksandr Valialkin	b3e93afd17	docs/CHANGELOG.md: add a link to Pushgateway protocol in the bugfix description for `74237ce5c0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692	2023-07-27 13:11:25 -07:00
Alexander Marshalov	74237ce5c0	fixed label values decoding for pushgateway compatibility (#4727 ) Fixed decoding of label values with slash for pushgateway and prometheus golang client compatibility + added some tests. (#4962)	2023-07-27 13:03:48 -07:00
Haleygo	3c297e0253	vmalert: add `keep_firing_for` field for alerting rule (#4669 ) vmalert: support `keep_firing_for` field for alerting rule https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4529 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-07-27 13:00:45 -07:00
hagen1778	46a80a2781	dependabot: return `schedule` field as it breaks validation on github side Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-27 12:59:28 -07:00
hagen1778	1250ebc063	vmalert: clarify docs for state restore with additional details The important change is to highlight that restore procedure happens only once and only for already loaded rules. Config hot-reload doesn't trigger the restore procedure. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-27 12:58:56 -07:00
hagen1778	91eddeca52	vmalert: revert accidental changes to Makefile rule Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-27 12:58:26 -07:00
Aliaksandr Valialkin	2e39070f95	vendor: run `make vendor-update`	2023-07-26 15:09:26 -07:00
Aliaksandr Valialkin	1b7d97787a	docs: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:40:06 -07:00
Aliaksandr Valialkin	98676a5223	docs/Cluster-VictoriaMetrics.md: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:13:24 -07:00
Aliaksandr Valialkin	da61f5dd34	docs/keyConcepts.md: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:12:46 -07:00
Aliaksandr Valialkin	5466183fd9	docs/Single-server-VictoriaMetrics.md: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:12:45 -07:00
Aliaksandr Valialkin	64ef01fccf	docs/Release-Guide.md: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin	3ef08d16fc	docs/Troubleshooting.md: use `1.` instead of `N.` in numbered bullets, so they are automatically adjusted by Github Markdown engine See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists	2023-07-26 14:12:32 -07:00
hagen1778	290443f7d1	docs: add bullet for gaps in timeseries to Troubleshooting https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3878 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3849 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3451 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin	3179d0528b	lib/promrelabel: return correct string representation for IfExpression containing a single selector This is a follow-up for `62651570bb` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635	2023-07-24 19:33:04 -07:00
Aliaksandr Valialkin	2d88ebd7cb	app/vmalert/datasource: substitute golang.org/x/exp/slices.SortFunc with sort.Slice This removes unnecessary third-party dependency on golang.org/x/exp. This is a follow-up for `da60a68d09` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945	2023-07-24 19:17:19 -07:00
Aliaksandr Valialkin	fa295c7daa	app/vmagent: add ability to shard outgoing data among multiple remote storage systems Add -remoteWrite.shardByURL command-line flag, which instructs vmagent to spread evenly outgoing time series data among the configured remote storage systems specified via -remoteWrite.url . Samples for the same time series go to the same -remoteWrite.url . This allows building horizontally scalable stream aggregation when samples for counter and histogram series must be aggregated by the same second-level vmagent instance. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637	2023-07-24 18:18:04 -07:00
Aliaksandr Valialkin	16c343f882	app/{vmselect,vlselect}/vmui: run `make vmui-update vmui-logs-update` after `b6ae325763`	2023-07-24 17:15:26 -07:00
Aliaksandr Valialkin	6e43664e24	lib/promrelabel: add support for a list of series selectors at IfExpression This makes possible specifying a list of series selectors at the following places: - Inside `if` option at relabeling rules - Inside `match` option at stream aggregation rules Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635	2023-07-24 17:09:59 -07:00
Aliaksandr Valialkin	c049778ad1	lib/streamaggr: follow-up for `736197179e` - Use a byte slice instead of a map for tracking indexes for matching series. This improves performance, since access by slice index is faster than access by map key. - Re-use the byte slice for tracking indexes for matching series. This removes unnecessary memory allocations and improves stream aggregation performance a bit. - Add an ability to return to the previous behvaiour by specifying -remoteWrite.streamAggr.dropInput command-line flag. In this case all the input samples are dropped when stream aggregation is enabled. - Backport the new stream aggregation behaviour from vmagent to single-node VictoriaMetrics when -streamAggr.config option is set. - Improve docs regarding this change at docs/CHANGELOG.md - Document the new behavior at docs/stream-aggregation.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4575	2023-07-24 17:06:09 -07:00
Zakhar Bessarab	470afac5ff	{lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag (#4575 ) * {lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag Changes default behaviour of keepInput flag to write series which did not match any aggregators to the remote write. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update app/vmagent/remotewrite/remotewrite.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-24 16:34:38 -07:00
Yury Molodov	9f440b1013	vmui: remove `monaco-editor` (#4698 ) * fix: remove monaco editor due to bundle size * vmui: update dependencies * fix: disable source map generation	2023-07-24 16:27:57 -07:00
hagen1778	42af241d59	docs: update datadog config recommendations Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-24 16:17:35 -07:00
Roman Khavronenko	08e66885ab	docs: mention lambdas and cloud functions for aggregation (#4705 ) * docs: mention lambdas and cloud functions for aggregation Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-24 16:16:43 -07:00
Haleygo	ee1b3a48e9	vmalert: unittest support stale datapoint (#4696 ) * vmalert: unittest support stale datapoint * add stale ut case	2023-07-24 16:15:27 -07:00
Roman Khavronenko	f536f22dfb	docs: add case for histograms aggregation (#4702 ) docs: add case for histograms aggregation Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-24 16:14:36 -07:00
Aliaksandr Valialkin	9d14c29667	lib/streamaggr: skip de-duplication for series, which do not match the configured aggregation rules Previously all the incoming samples were de-duplicated, even if their series doesn't match aggregation rule filters. This could result in increased CPU usage. Now the de-duplication isn't applied to samples for series, which do not match aggregation rule filters. Such samples are just ignored.	2023-07-22 16:46:17 -07:00
Zakhar Bessarab	d800545055	app/vlinsert/loki: manually remove bloat dependecies for generate proto file (#4686 ) Co-authored-by: f41gh7 <nik@victoriametrics.com>	2023-07-22 15:37:30 -07:00
Nikolay	30b32583f4	lib/storage: pre-create timeseries before indexDB rotation (#4652 ) * lib/storage: pre-create timeseries before indexDB rotation during an hour before indexDB rotation start creating records at the next indexDB it must improve performance during switch for the next indexDB and remove ingestion issues. Since there is no need for creation new index records for timeseries already ingested into current indexDB https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 * lib/storage: further work on indexdb rotation optimization - Document the change at docs/CHAGNELOG.md - Move back various caches from indexDB to Storage. This makes the change less intrusive. The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID) entries for both the current and the next indexDB. - Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function. This improves code readability and maintainability a bit. - Rewrite and simplify the code responsible for calculating the next retention timestamp. Add various tests for corner cases of this code. - Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called. It is OK to add indexdb entries on demand in this function. This simplifies the code. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 * docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-22 15:23:14 -07:00
yrka	7750c5a553	Update MetricsQL.md (#4689 ) misspell	2023-07-22 14:19:30 -07:00
Zakhar Bessarab	2fe33b3d97	app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687 ) * app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries Fixes construction of URL parameters for graphite render to allow overriding "from" parameter. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmalert/datasource/graphite: update flow for building URL parameters Makes flow of building URL parameters same as Prometheus datasource has: 1) Setting all default values 2) Merging those values with provided `extraParams` Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/CHANGELOG.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-07-22 14:18:52 -07:00
Roman Khavronenko	af56dff7ce	dependabot: disable version update for packages (#4684 ) The change disables version updates for repo packages. Please note, security updates should not be affected by the change according to https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#open-pull-requests-limit: ``` open-pull-requests-limit By default, Dependabot opens a maximum of five pull requests for version updates. Once there are five open pull requests from Dependabot, Dependabot will not open any new requests until some of those open requests are merged or closed. This option has no impact on security updates, which have a separate, internal limit of ten open pull requests. ``` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-22 14:17:50 -07:00
Aliaksandr Valialkin	1ce82f874c	lib/streamaggr: follow up for `70773f53d7` - Round staleness_interval durations to the upper number of seconds. This should prevent from under-calculations for fractional staleness intervals. - Rename stalenessInterval field at *AggrState structs into stalenessSecs, since it holds seconds. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667	2023-07-20 21:56:36 -07:00
Aliaksandr Valialkin	cbc5367812	docs/VictoriaLogs/data-ingestion: mention that VictoriaLogs automatically extracts message and timestamp from data ingested via Loki JSON API	2023-07-20 21:55:46 -07:00
Aliaksandr Valialkin	8c59813c17	app/vlinsert/loki: fix build for architectures where int is 32-bit	2023-07-20 21:55:28 -07:00
Aliaksandr Valialkin	9e6cc3c495	deployment: update VictoriaLogs Docker image tag from from v0.2.0-victorialogs to v0.3.0-victorialogs	2023-07-20 21:55:09 -07:00
Aliaksandr Valialkin	dcf5b42670	lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer. This is a follow-up for `140e7b6b74`	2023-07-20 21:54:51 -07:00
Aliaksandr Valialkin	c94a5d95a2	docs/VictoriaLogs/CHANGELOG.md: cut v0.3.0-victorialogs	2023-07-20 21:54:28 -07:00
Aliaksandr Valialkin	324a3c5288	lib/promscrape: follow-up after `6aa50ca954` - Improve docs - Hide `debug relabeling` column when -promscrape.dropOriginalLabels command-line flag is set - Inline the code from the added template functions, since the code is harder to follow with the template functions, especially when these functions have misleading names. Also, these functions are used only in one place, e.g. they do not reduce the amounts of code. - Hide `click to show original labels` title at `labels` column when original labels aren't available. - Show the reason on whey original labels aren't available at /service-discovery page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597	2023-07-20 21:54:09 -07:00
Aliaksandr Valialkin	c921bc0833	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update` after recent changes to VMUI Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4294	2023-07-20 21:53:51 -07:00
Yury Molodov	7757a31a73	fix: change getting serverUrl for vmui-logs (#4604 ) (#4677 )	2023-07-20 21:53:30 -07:00
Yury Molodov	02a182e2a1	feat: change columns for active queries (#4676 )	2023-07-20 21:53:12 -07:00
Yury Molodov	15e1d16afc	vmui: enhancements multiline field editing (#4294 ) * fix: change textarea for relabel page * feat: add comment for monaco theme * fix: change behavior of multiline fields * vmui: merge master	2023-07-20 21:52:53 -07:00
Aliaksandr Valialkin	f086a5ba63	docs/VictoriaLogs/CHANGELOG.md: document the addition of Grafana Loki push protocol support (aka Promtail) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482	2023-07-20 21:52:33 -07:00
Aliaksandr Valialkin	30098ac8bd	app/vlinsert/loki: follow-up after `09df5b66fd` - Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki - Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header - Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler - Check JSON field types more strictly. - Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients, which store timestamps in float64 instead of int64. - Optimize parsing of Loki labels in Prometheus text exposition format. - Simplify tests. - Remove lib/slicesutil, since there are no more users for it. - Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels as stream fields in most Loki setups. - Allow empty of missing timestamps in the ingested logs. The current timestamp at VictoriaLogs side is then used for the ingested logs. This simplifies debugging and testing of the provided HTTP-based data ingestion APIs. The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 . This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation similar to the one used at lib/prompb or lib/prompbmarshal .	2023-07-20 21:52:11 -07:00
Dmytro Kozlov	41f0227af2	docs: update information about tsdb usage in cluster version (#4678 ) update information about tsdb usage in cluster version	2023-07-20 21:50:58 -07:00
hagen1778	1d2a0e0e10	docs: fix the next release version for vmalert Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-20 21:49:34 -07:00
Zakhar Bessarab	5b3000f7e8	docs: fix broken link to managed VM (#4679 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-20 21:49:06 -07:00
hagen1778	5b963e1971	docs: mention streaming aggregation as way to reduce cardinality Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-20 21:48:09 -07:00
Alexander Marshalov	9ba03b4838	allow configuring staleness interval in stream aggregation (#4667 ) (#4670 ) --------- Signed-off-by: Alexander Marshalov <_@marshalov.org> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-07-20 21:47:29 -07:00
hagen1778	f7d60613a9	docs: specify min version and limitations for vmalert's unit tests Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-20 21:27:50 -07:00
Aliaksandr Valialkin	a0b7def89d	app/vmselect/promql: fix tests after `781947a7e2`	2023-07-20 21:25:30 -07:00
Haleygo	939c8b8372	vmalert: init unit test (#4596 ) vmalert: support unit tests See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-07-20 21:19:45 -07:00
hagen1778	a0cc6eb7a1	docs: mention the simplest way to migrate data in vmctl docs Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-20 21:12:18 -07:00
Dmytro Kozlov	f0d8f77e6d	app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used (#4616 ) * app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used * app/vmagent: hide links if OriginalLabels was dropped * app/vmagent: update CHANGELOG.md and added information to the docs * app/vmagent: fix comments	2023-07-20 19:21:41 -07:00
Zakhar Bessarab	7e24edd506	app/vlinsert/loki: fix compatibility with latest MetricsQL lib (#4675 ) Loki uses default labels format without "or" operator. This format can't create a list of LabelFilters, so only first set of LabelFilters should be used. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-20 17:06:09 -07:00
Zakhar Bessarab	5b3cbd4db1	app/vlinsert: add support of loki push protocol (#4482 ) * app/vlinsert: add support of loki push protocol - implemented loki push protocol for both Protobuf and JSON formats - added examples in documentation - added example docker-compose Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vlinsert: move protobuf metric into its own file Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker/victorialogs/promtail: update reference to docker image Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker/victorialogs/promtail: make volume name unique Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vlinsert/loki: add license reference Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/docker/victorialogs/promtail: fix volume name Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vlinsert/loki: move entities to places where those are used Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vlinsert/loki: refactor to use common components - use CommonParameters from insertutils - stop ingestion after first error similar to elasticsearch and jsonline Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vlinsert/loki: address review feedback - add missing logstorage.PutLogRows calls - refactor tenant ID parsing to use common function - reduce number of allocations for parsing by reusing logfields slices - add tests and benchmarks for requests processing funcs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-20 16:49:43 -07:00
Aliaksandr Valialkin	5ec1ec39cd	docs/stream-aggregation.md: mention that sum_samples, stddev, stdvar, histogram_bucket and quantiles outputs must be applied only to gauge metrics	2023-07-20 11:31:38 -07:00
Aliaksandr Valialkin	0cbe5ccb4a	app/vmselect: rename promql.WriteActiveQueries() to promql.ActiveQueriesHandler() This makes it more consistent with the rest of handlers inside app/vmselect/main.go This is a follow-up for `6a96fd8ed5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4598	2023-07-20 11:30:40 -07:00
Aliaksandr Valialkin	992c300ce9	all: replace atomic.Value with atomic.Pointer[T] This eliminates the need in .(*T) casting for results obtained from Load() Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map, because map is already a pointer type.	2023-07-19 17:48:26 -07:00
Roman Khavronenko	92cd98e366	docs: mention tags routing for new release (#4566 ) * docs: mention tags routing for new release Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update docs/Release-Guide.md --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-19 16:56:04 -07:00
Yury Molodov	44b4963ac6	feat: optimize vmui-log bundle size (#4602 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-19 16:52:18 -07:00
Aliaksandr Valialkin	443c266406	app/vmalert/README.md: sync with docs/vmalert.md after `54b7bd4564`	2023-07-19 16:31:30 -07:00
Aliaksandr Valialkin	29c6bcbd5b	app/vmselect: follow-up after `6a96fd8ed5` - Add `Active queries` chapter to VMUI docs - Set `Content-Type: json` header inside promql.WriteActiveQueries() handler, in order to be consistent with other request handlers called at app/vmselect/main.go - Pass the request to promql.WriteActiveQueries() handler, so it can change its output depending on the provided request params. This also improves consistency of promql.WriteActiveQueries() args with other request hanlers at app/vmselect/main.go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4653	2023-07-19 16:31:26 -07:00
Yury Molodov	3ad80e281f	vmui: add Active Queries page (#4653 ) * feat: add page to display a list of active queries (#4598) * app/vmagent: code formatting * fix: remove console --------- Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>	2023-07-19 16:02:58 -07:00
dependabot[bot]	9d55da5d26	build(deps-dev): bump word-wrap in /app/vmui/packages/vmui (#4664 ) Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4. - [Release notes](https://github.com/jonschlinkert/word-wrap/releases) - [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4) --- updated-dependencies: - dependency-name: word-wrap dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-19 15:43:59 -07:00
Aliaksandr Valialkin	a3c8f902c1	vendor: update github.com/VictoriaMetrics/metricsql from v0.60.0 to v0.61.1 This adds support for passing durations via WITH template vars: - `WITH (w = 5m) m[w]` is transformed to `m[5m]` - `WITH (f(w, step, off) = m[w:step] offset off) f(5m, 10s, 1h)` is transformed to `m[5m:10s] offset 1h` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4025 Updates https://github.com/VictoriaMetrics/metricsql/issues/12 See also the initial implementation by @lujiajing1126 at https://github.com/VictoriaMetrics/metricsql/pull/13	2023-07-19 15:04:38 -07:00
Haleygo	1794a97ebe	add vmalertmanager filter for health alerts (#4665 )	2023-07-19 14:50:06 -07:00
hagen1778	7703059746	docs: typo Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-19 14:49:31 -07:00
Artem Navoiev	a600cc3645	specify formating in vmalert doc Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-19 14:49:02 -07:00
Roman Khavronenko	80768d53dd	docs: follow-up after `aec4b5db81` (#4638 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-19 14:48:17 -07:00
Aliaksandr Valialkin	4fb505dfe9	vendor: update github.com/VictoriaMetrics/metricsql from v0.59.1 to v0.60.0 This update adds support for metricsql.Prettify() function, which is going to be used for prettifying long MetricsQL queries in VMUI	2023-07-18 19:19:33 -07:00
Aliaksandr Valialkin	31b7c947ef	vendor: `make vendor-update`	2023-07-18 16:14:07 -07:00
Aliaksandr Valialkin	798ff17c42	properly encode the right part of binary operation if it starts with `prefix` series selector	2023-07-18 16:03:57 -07:00
Aliaksandr Valialkin	5819d4e6f7	lib/logstorage: properly encode `"offset"` search word just after _time filter	2023-07-18 16:03:57 -07:00
Roman Khavronenko	debe1793bb	vmalert: follow-up after `d4ac4b7813` (#4659 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-18 16:03:28 -07:00
venkatbvc	bd2a37429c	vmalert: allow to blackhole alerting notifications (#4639 ) vmalert: support option to blackhole alerting notifications https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4122 --------- Co-authored-by: Rao, B V Chalapathi <b_v_chalapathi.rao@nokia.com>	2023-07-18 16:02:48 -07:00
hagen1778	456a2e70fd	docs: mention change from `6f3fee197e` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-18 16:01:59 -07:00
Yury Molodov	cde32da04f	vmui: add tip to Explore Metrics page (#4615 ) * feat: add tip to Explore Metrics page (#4248) * fix: update description page	2023-07-18 16:01:24 -07:00
Zakhar Bessarab	0a8d39e0d5	dashboards/cluster: fix using storage filter for cache usage panel (#4657 ) Using `job=~$job_storage` forces "Cache usage" panel to display only vmstorage caches, but there is a cache peresent at vmselect(`promql/rollupResult`). Updated selector to match generic `$job` so that all caches will be displayed with an option to display per-job caches. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-18 16:00:44 -07:00
Aliaksandr Valialkin	2042ea7bf3	docs/VictoriaLogs/CHANGELOG.md: remove newlines inside FEATURE descriptions, since they dont play nicely when copy-n-pasting to https://github.com/VictoriaMetrics/VictoriaMetrics/releases/	2023-07-17 23:31:37 -07:00
Aliaksandr Valialkin	15ccde8dd3	all: use the new release for VictoriaLogs - v0.2.0-victorialogs	2023-07-17 23:20:07 -07:00
Aliaksandr Valialkin	8e42b2294c	docs/VictoriaLogs: add CHANGELOG.md	2023-07-17 23:14:23 -07:00
dependabot[bot]	6508659252	build(deps): bump github.com/cheggaaa/pb/v3 from 3.1.3 to 3.1.4 (#4645 ) Bumps [github.com/cheggaaa/pb/v3](https://github.com/cheggaaa/pb) from 3.1.3 to 3.1.4. - [Commits](https://github.com/cheggaaa/pb/compare/v3.1.3...v3.1.4) --- updated-dependencies: - dependency-name: github.com/cheggaaa/pb/v3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-17 19:08:11 -07:00
Aliaksandr Valialkin	da2ef397fa	lib/logstorage: add abilty to speficy offset for the selected _time filter The following syntax is supported: _time:filter offset off For example: - _time:5m offset 1h - 5-minute duration one hour before the current time - _time:2023 offset 2w - 2023 year with the 2 weeks offset in the past	2023-07-17 19:07:14 -07:00
Aliaksandr Valialkin	e1f7e0b455	lib/logstorage: log the -retentionPeriod and -futureRetention values when the ingested log entry has timestamp outside the configured retention This should simplify debugging	2023-07-17 18:23:45 -07:00
Aliaksandr Valialkin	6751a08071	lib/logstorage: support for short form of _time:(now-duration, now] filter: _time:duration	2023-07-17 18:23:43 -07:00
Aliaksandr Valialkin	8fdfd13a29	lib/logstorage: LogsQL: replace exact_prefix("...") with exact("...") This makes LogsQL queries more consistent with i("...") and i("...") syntax	2023-07-17 17:19:45 -07:00
Aliaksandr Valialkin	5ace0701d3	app/vmselect/promql: add the ability to copy all the labels from `one` side of group_left()/group_right() operation This is performed by specifying `` inside group_left()/group_right(). Also allow specifying prefix for the copied labels via `group_left(...) prefix "..."` and `group_right(...) prefix "..."` syntax. For example, the following query adds all the namespace-related labels to pod info, and prefixes all the copied label names with "ns_" prefix: kube_pod_info on(namespace) group_left(*) prefix "ns_" kube_namespace_labels This resolves the following StackOverflow questions: - https://stackoverflow.com/questions/76661818/how-to-add-namespace-labels-to-pod-labels-in-prometheus - https://stackoverflow.com/questions/76653997/how-can-i-make-a-new-copy-of-kube-namespace-labels-metric-with-a-different-name	2023-07-17 16:58:30 -07:00
Aliaksandr Valialkin	cc54fa2a56	app/vmselect/promql: recommend to use `(a op b) keep_metric_names` instead of `a op b keep_metric_names` The `a op b keep_metric_names` is ambigouos to `a op (b keep_metric_names)` when `b` is a transform or rollup function. For example, `a + rate(b) keep_metric_names`. So it is better to use more clear syntax: `(a op b) keep_metric_names` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3710	2023-07-16 23:47:15 -07:00
Aliaksandr Valialkin	ec375e2e15	docs/keyConcepts.md: clarify that the number of `,`-delimited filters per each `or` group can be arbitrary Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997	2023-07-16 23:47:15 -07:00
Aliaksandr Valialkin	8715e331c8	docs/VictoriaLogs/data-ingestion/README.md: document how to verify whether the data is successfully ingested into VictoriaLogs	2023-07-16 22:54:05 -07:00
Zakhar Bessarab	781947a7e2	metricsql: add support of using keep_metric_names for binary operations (#4109 ) * metricsql: add support of using keep_metric_names for binary operations This should help to avoid confusion with queries like one in the issue #3710. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * wip --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-07-16 03:01:27 -07:00
Artem Navoiev	30e84f4c66	docs: logs disable table of content for pages with no h2-h6, add aliases Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-16 00:08:06 -07:00
Artem Navoiev	6363a7bb80	docs: victorialogs add front-matter for readme pages Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-16 00:08:06 -07:00
Artem Navoiev	2963cbbb74	docs: victorialogs add front-matter for data-ingestion items Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-16 00:08:06 -07:00
Aliaksandr Valialkin	a7fdc3fcc7	all: add support for `or` filters in series selectors This commit adds ability to select series matching distinct filters via a single series selector. For example, the following selector selects series with either {env="prod",job="a"} or {env="dev",job="b"} labels: {env="prod",job="a" or env="dev",job="b"} The `or` filter is supported in all the VictoriaMetrics tools now. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997 Uses https://github.com/VictoriaMetrics/metricsql/pull/14	2023-07-15 23:56:18 -07:00
Aliaksandr Valialkin	6a65af5112	all: replace ElasticSearch -> Elasticsearch for the sake of consistency This is a follow-up for `7f6b5dc47b`	2023-07-14 10:52:43 -07:00
Aliaksandr Valialkin	d721109961	docs/CHANGELOG.md: sync with master branch	2023-07-14 10:48:40 -07:00
Aliaksandr Valialkin	99ec7ef28f	docs/VictoriaLogs: use exact number when comparing efficiency of VictoriaLogs and Elasticsearch	2023-07-14 10:47:29 -07:00
Haleygo	5e5c805599	vmalert: fix evalTS after modify group interval (#4629 )	2023-07-14 10:47:29 -07:00
subham sarkar	9d3cfc03cd	docs: Update README.md (#4613 ) Fix grammatical mistakes and also s/ElasticSearch/Elasticsearch	2023-07-14 10:46:13 -07:00
Artem Navoiev	14d21cfc73	githubaciton: sync-docs change set output code as current one will be deprecated soon Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	c245e38f8d	fix alias Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	6b98511ac9	add front-matter for some of victorialogs pages Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	4c78a9cc90	Update .github/workflows/sync-docs.yml Co-authored-by: Max Golionko <8kirk8@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	ec936262d1	enable only when changes in docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	bbc9e349bd	moar debug Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	f85611859c	fix path to main repo Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	abc737747e	more debug Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	64718a3a49	move debug Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	953f146091	remove env Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	8cad820526	debug workflow1 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	f01dea5728	add workdir to gpg step Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:44:04 -07:00
Artem Navoiev	c75202657c	add job to sync the docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-14 10:39:11 -07:00
Roman Khavronenko	89b5a6a4d5	vmctl: mention replicationFactor during migration (#4633 ) Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4624 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-14 10:36:46 -07:00
Aliaksandr Valialkin	3d23fd9853	lib/storage: move series registration in caches from createAllIndexesForMetricName into a separate function - putSeriesToCache This makes the code more clear and easier to read This is a follow-up for `7094fa38bc`	2023-07-13 23:17:14 -07:00
dependabot[bot]	a15a66ee89	build(deps): bump tough-cookie in /app/vmui/packages/vmui (#4603 ) Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.1.2 to 4.1.3. - [Release notes](https://github.com/salesforce/tough-cookie/releases) - [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md) - [Commits](https://github.com/salesforce/tough-cookie/compare/v4.1.2...v4.1.3) --- updated-dependencies: - dependency-name: tough-cookie dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-13 22:22:53 -07:00
Aliaksandr Valialkin	4b86522f4c	lib/mergeset: skip common prefix in binarySearchKey() function This should improve performance a bit when the search if performed among items with long common prefix	2023-07-13 22:05:14 -07:00
Aliaksandr Valialkin	203a436066	lib/storage: optimize BenchmarkIndexDBGetTSIDs() - Sort MetricName tags only once before the benchmark loop. - Obtain indexSearch per each benchmark loop in order to give a chance for background merge for the recently created parts	2023-07-13 21:49:54 -07:00
Aliaksandr Valialkin	fbddb4ad32	lib/storage: typo fix after `e1cf962bad` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401	2023-07-13 21:29:02 -07:00
Aliaksandr Valialkin	7d359d17d1	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:34 -07:00
Aliaksandr Valialkin	e1cf962bad	lib/storage: switch from global to per-day index for `MetricName -> TSID` mapping Previously all the newly ingested time series were registered in global `MetricName -> TSID` index. This index was used during data ingestion for locating the TSID (internal series id) for the given canonical metric name (the canonical metric name consists of metric name plus all its labels sorted by label names). The `MetricName -> TSID` index is stored on disk in order to make sure that the data isn't lost on VictoriaMetrics restart or unclean shutdown. The lookup in this index is relatively slow, since VictoriaMetrics needs to read the corresponding data block from disk, unpack it, put the unpacked block into `indexdb/dataBlocks` cache, and then search for the given `MetricName -> TSID` entry there. So VictoriaMetrics uses in-memory cache for speeding up the lookup for active time series. This cache is named `storage/tsid`. If this cache capacity is enough for all the currently ingested active time series, then VictoriaMetrics works fast, since it doesn't need to read the data from disk. VictoriaMetrics starts reading data from `MetricName -> TSID` on-disk index in the following cases: - If `storage/tsid` cache capacity isn't enough for active time series. Then just increase available memory for VictoriaMetrics or reduce the number of active time series ingested into VictoriaMetrics. - If new time series is ingested into VictoriaMetrics. In this case it cannot find the needed entry in the `storage/tsid` cache, so it needs to consult on-disk `MetricName -> TSID` index, since it doesn't know that the index has no the corresponding entry too. This is a typical event under high churn rate, when old time series are constantly substituted with new time series. Reading the data from `MetricName -> TSID` index is slow, so inserts, which lead to reading this index, are counted as slow inserts, and they can be monitored via `vm_slow_row_inserts_total` metric exposed by VictoriaMetrics. Prior to this commit the `MetricName -> TSID` index was global, e.g. it contained entries sorted by `MetricName` for all the time series ever ingested into VictoriaMetrics during the configured -retentionPeriod. This index can become very large under high churn rate and long retention. VictoriaMetrics caches data from this index in `indexdb/dataBlocks` in-memory cache for speeding up index lookups. The `indexdb/dataBlocks` cache may occupy significant share of available memory for storing recently accessed blocks at `MetricName -> TSID` index when searching for newly ingested time series. This commit switches from global `MetricName -> TSID` index to per-day index. This allows significantly reducing the amounts of data, which needs to be cached in `indexdb/dataBlocks`, since now VictoriaMetrics consults only the index for the current day when new time series is ingested into it. The downside of this change is increased indexdb size on disk for workloads without high churn rate, e.g. with static time series, which do no change over time, since now VictoriaMetrics needs to store identical `MetricName -> TSID` entries for static time series for every day. This change removes an optimization for reducing CPU and disk IO spikes at indexdb rotation, since it didn't work correctly - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 . At the same time the change fixes the issue, which could result in lost access to time series, which stop receving new samples during the first hour after indexdb rotation - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698 The issue with the increased CPU and disk IO usage during indexdb rotation will be addressed in a separate commit according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401#issuecomment-1553488685 This is a follow-up for `1f28b46ae9`	2023-07-13 17:03:50 -07:00
Aliaksandr Valialkin	1bce67df06	lib/storage: fix possible test failure in TestStorageAddRowsConcurrent The number of parts in the snapshot partition may be zero if concurrent goroutine just started creating new partition, but didn't put data into it yet when the current goroutine made a snapshot.	2023-07-13 15:03:51 -07:00
Aliaksandr Valialkin	733032e514	lib/mergeset: simplify fulsuhInmemoryParts() a bit	2023-07-13 12:33:43 -07:00
Aliaksandr Valialkin	df67b78f75	docs/CHANGELOG.md: clarify the description of the bugfix at `177a0c1ca9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4555	2023-07-13 12:19:00 -07:00
Aliaksandr Valialkin	3e5e55d235	Revert "make: add goimports task (#4582 )" This reverts commit `20b18e9feb`. Reason for revert: running goimports on `make check-all` introduces the following issues: - It runs only on modified files, which weren't commited yet into git repository. This means the formatting for the remaining files becomes different comparing to the formatting for the changed files. This also means that the goimports has no any effect at github actions and when the changed code is already commited to git repository. - `gomiports` performs formatting in the same way as gofmt, so `make fmt` becomes unnecessary. But when `gofmt` is substituted with `goimports`, then it performs unnecessary formatting for .qtpl.go files. It is possible to make a hack, which will prepare a list of all the .go files at lib/ and app/ without the *.qtpl.go files, and then feed this list to `goimports`, but this looks too fragile for the task of just fixing the ordering of Go imports. So it is better to leave source code formatting as is with `gofmt`, while manually fixing improper ordering of Go import from time to time in dedicated commits until better solution arises.	2023-07-13 12:18:02 -07:00
Zakhar Bessarab	64149782ac	make: add goimports task (#4582 ) * make: add goimports task Adds task to fix imports formatting implace. Formats imports into: - native library - external libraries - local packages based on github.com/VictoriaMetrics/VictoriaMetrics prefix Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * make: add goimports install task Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * make: run goimports only for changed files Applying goimports to all existing files would create a lot of problems with cherry-picking changes between different branches used for development. To avoid this it was decided to only run goimports on changed files to fix formatting gradually. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * make: update goimports to run on all changed files Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-13 12:17:31 -07:00
Aliaksandr Valialkin	650af7c5ca	app/vmalert: silence golagci-lint at TestAlertingRule_Template Add a break if gotAlert is nil This removes the following golangci-lint warning: app/vmalert/alerting_test.go:868:8: SA5011(related information): this check suggests that the pointer can be nil (staticcheck) if gotAlert == nil { ^	2023-07-13 12:16:00 -07:00
Dmytro Kozlov	f31ac064f9	app/vmctl: fix panic `--remote-read-filter-time-start` flag not defined (#4605 ) * app/vmctl: fix panic `--remote-read-filter-time-start` flag not defined * app/vmctl: update CHANGELOG.md --------- Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-07-13 12:13:21 -07:00
Dmytro Kozlov	555a0a9d57	app/vmctl: fix issue with adding many seconds (#4617 ) * app/vmctl: fix issue with adding many seconds * app/vmagent: add CHANGELOG.md	2023-07-13 12:09:54 -07:00
Roman Khavronenko	fdccb56620	vmalert: check for negative offset for missed rounds (#4628 ) It could happen for low evaluation intervals and irregular delays during execution that evaluation time would get a negative offset. This could result into cumulative discrepancy between the actual time and evaluation time for rules. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-13 12:05:52 -07:00
Dmytro Kozlov	3d0f846a79	lib/logstorage: fix panic (#4620 )	2023-07-13 12:04:59 -07:00
Alexander Marshalov	09cc17a6b1	fixed typo (#4622 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-13 11:29:57 -07:00
Alexander Marshalov	7a5e5f6a89	add info about using stream aggregation as statsd alternative (#4600 ) (#4621 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-13 11:29:57 -07:00
Aliaksandr Valialkin	d8b8fc0343	lib/logstorage: fix TestValuesEncoder() on 32-bit architectures	2023-07-13 11:28:04 -07:00
Aliaksandr Valialkin	2636c35cec	docs/VictoriaLogs/FAQ.md: small fixes	2023-07-12 01:10:40 -07:00
Aliaksandr Valialkin	b07a1c85b9	all: update Go builder from 1.20.5 to 1.20.6 See https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved	2023-07-12 01:00:24 -07:00
Aliaksandr Valialkin	bcea7a0a92	docs/keyConcepts.md: cosmetic fixes after `b67bd156d5`	2023-07-12 00:30:33 -07:00
Aliaksandr Valialkin	b99fcb7d7a	docs/VictoriaLogs: add FAQ	2023-07-12 00:30:33 -07:00
Aliaksandr Valialkin	f8299ee9d8	docs/VictoriaLogs/README.md: make it clear that VictoriaLogs is open source	2023-07-12 00:30:33 -07:00
Alexander Marshalov	8277b64ec3	follow up for #4612 and #4584 (#4614 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-12 00:30:06 -07:00
Zakhar Bessarab	a667bdaad5	doc: fix image src after `b67bd156` (#4612 ) Followup for `b67bd156d5` Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-12 00:29:36 -07:00
Alexander Marshalov	9e8c02520c	added info about `search.latencyOffset` to key concepts (#4567 ) (#4584 ) Signed-off-by: Alexander Marshalov <_@marshalov.org> Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-12 00:25:32 -07:00
Aliaksandr Valialkin	a5f55259f6	docs/VictoriaLogs: make more prominent the information about returned log fields in query responses Thanks to @candlerb for suggestions on how to improve VictoriaLogs docs at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4609#issuecomment-1629758426	2023-07-10 15:01:54 -07:00
Dmytro Kozlov	5c4ca4aea8	app/vmctl: remove undefined flag from the documentation. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4552 . (#4606 )	2023-07-10 15:01:54 -07:00
Aliaksandr Valialkin	f65153018b	app/{vmselect,vlselect}: run `make vmui-update vmui-logs-update`	2023-07-09 12:44:04 -07:00
Aliaksandr Valialkin	81fe089546	docs/Single-server-VictoriaMetrics.md: mention how to use Prometheus config file with unsupported options Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4274	2023-07-09 12:37:44 -07:00
Aliaksandr Valialkin	371036eb35	docs/VictoriaLogs: small clarifications	2023-07-09 12:36:38 -07:00
Zakhar Bessarab	ddd918b93c	docs: make `httpAuth.` flags description less ambiguous (#4588 ) docs: make `httpAuth.` flags description less ambiguous Currently, it may confuse users whether `httpAuth.` flags are used by HTTP client or server configuration(see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4586 for example). Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * docs: fix a typo Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-09 12:36:14 -07:00
Max Golionko	e55a2b4137	ci: fix sandbox update job (#4549 ) Prevent sandbox deploy job run if tag name is empty.	2023-07-09 12:34:46 -07:00
Haleygo	ef8e3eb9b3	vmselect: fix result in Prometheus query when time is small (#4578 ) vmselect: fix result in Prometheus query when time is small Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-07-09 12:33:29 -07:00
Aliaksandr Valialkin	e1a2404db5	app/vmselect/netstorage: follow-up after `173ccf4333` - Clarify docs about -replicationFactor command-line flag at vmselect - Clarify description for -replicationFactor and -search.skipSlowReplicas command-line flags - Fix the logic for returning responses if -search.skipSlowReplicas command-line flag is enabled. The logic was broken in the `173ccf4333`, so it could return responses only if some of vmstorage nodes return error, while it should return when query results are successfully collected from more than (len(storageNodes) - replicationFactor) vmstorage nodes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2023-07-09 11:58:22 -07:00
Haleygo	3c2308fd52	vmalert:fix query request using rfc3339 format (#4577 ) vmalert: consistently use time.RFC3339 format for time in queries Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-07-09 11:03:10 -07:00
Haleygo	14e242d0b9	vmselect: fix result collect count (#4599 )	2023-07-08 08:21:27 +02:00
Roman Khavronenko	173ccf4333	vmselect: introduce `search.skipSlowReplicas` cmd-line flag (#4538 ) * vmselect: introduce `search.skipSlowReplicas` cmd-line flag vmselect has two logical conditions during request processing when `-replicationFactor` cmd-line flag is set: 1. If at least `len(storageNodes) - replicationFactor` responded, it could skip waiting for the rest of nodes to respond. This could lead to problems described here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207. 2. Mark response as partial if less than `len(storageNodes) - replicationFactor` responded without an error. The P1 showed itself error-prone and became the main reason why `-replicationFactor` wasn't recommended to use at vmselect level. However, this optimization could be still very useful in situations when there are slow and fast replicas in cluster. But P2 remains viable and important conditionless. Hiding P1 behind the feature-flag `search.skipSlowReplicas` should make `-replicationFactor` flag usable again. And let users choose whether they want P1 to be respected. Related issues https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711 Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: update changelog Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-07 11:50:26 +02:00
Aliaksandr Valialkin	e2a2d64053	vendor: run `make vendor-update`	2023-07-07 00:05:50 -07:00
Artem Navoiev	bf49efc11a	update logo width in cluster doc to 300 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 23:16:42 -07:00
Roman Khavronenko	109e55f865	vmalert: allow disabling of `step` param attached to instant queries (#4574 ) vmalert: allow disabling of `step` param attached to instant queries This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4573 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 23:13:56 -07:00
Aliaksandr Valialkin	0107d78639	docs/vmgateway.md: update `-help` output	2023-07-06 23:07:47 -07:00
Aliaksandr Valialkin	72dd0b9fac	docs/Cluster-VictoriaMetrics.md: update `-help` output	2023-07-06 23:06:11 -07:00
Aliaksandr Valialkin	ee4280d132	docs/vmbackupmanager.md: update `-help` output	2023-07-06 22:57:31 -07:00
Aliaksandr Valialkin	921d8b36b5	docs/vmrestore.md: update `-help` output	2023-07-06 22:55:26 -07:00
Aliaksandr Valialkin	e1993dadc2	docs/vmbackup.md: update `-help` output	2023-07-06 22:54:15 -07:00
Aliaksandr Valialkin	e35abdd2e4	docs/vmauth.md: update `-help` output	2023-07-06 22:52:48 -07:00
Aliaksandr Valialkin	316abe550d	docs/vmalert.md: update `-help` output	2023-07-06 22:50:47 -07:00
Aliaksandr Valialkin	b9790515e4	docs/vmagent.md: update `-help` output	2023-07-06 22:48:23 -07:00
Aliaksandr Valialkin	65d7194588	docs/Single-server-VictoriaMetrics.md: update `-help` output	2023-07-06 22:45:58 -07:00
Aliaksandr Valialkin	eea088d87f	docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix This is a follow-up for `5eb5df96e2`	2023-07-06 22:42:02 -07:00
Aliaksandr Valialkin	eeb53660b8	docs/CHANGELOG.md: use the proper link to the issue related to the commit `7a92263459` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4402	2023-07-06 22:41:43 -07:00
Aliaksandr Valialkin	67a8992798	docs/CHANGELOG.md: remove redundant info from the url to consulagent_sd_configs docs This is a follow-up for `40d12be607`	2023-07-06 22:41:23 -07:00
Aliaksandr Valialkin	40f1ccba67	docs/CHANGELOG.md: clarify the description of the bugfix at `ce7141383d`	2023-07-06 22:41:03 -07:00
Aliaksandr Valialkin	dc89e1f644	app/vmselect/graphite: follow-up after `c7884f8686` - Consistently use -search.maxGraphiteTagValues for limiting tag values from auto-complete API - Use -search.maxGraphiteSeries for limiting paths (aka series), which can be returned from Graphite series API - Clarify the change in docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841	2023-07-06 22:33:30 -07:00
Alexander Marshalov	eb611c3dc3	fix removing storage data dir before restoring from backup (#598 ) * fix removing storage data dir before restoring from backup Signed-off-by: Alexander Marshalov <_@marshalov.org> * fix review comment Signed-off-by: Alexander Marshalov <_@marshalov.org> * fix review comment Signed-off-by: Alexander Marshalov <_@marshalov.org> * fixes after merge with `enterprise-single-node` branch Signed-off-by: Alexander Marshalov <_@marshalov.org> --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-06 22:32:12 -07:00
Aliaksandr Valialkin	eda26a8352	lib/backup/actions: remove misleading comment about the default value for Concurrency field	2023-07-06 22:31:40 -07:00
Aliaksandr Valialkin	2f19ba0f75	app/vmselect/netstorage: follow-up after `11ac551d52` - Clarify the scope of the fix at docs/CHANGELOG.md - Handle the case when -search.maxSamplesPerSeries limit is exceeded in the same way as the -search.maxSamplesPerQuery limit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472	2023-07-06 22:26:47 -07:00
Roman Khavronenko	690f58c016	docs: explicitly mention errors processing for import APIs (#4583 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:26:04 -07:00
Denys Holius	97a5bdf4f0	docs: adds curl commands to clear the query cache (#4468 ) adds curl commands to clear query cache on vmselect/VM Single	2023-07-06 22:25:21 -07:00
Aliaksandr Valialkin	85c134feec	docs/VictoriaLogs/LogsQL.md: various fixes according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4579	2023-07-06 22:24:41 -07:00
Aliaksandr Valialkin	ebd08cd822	lib/logstorage: go fmt	2023-07-06 22:24:18 -07:00
Aliaksandr Valialkin	7cf5efc5b8	README.md: add a link to VictoriaLogs	2023-07-06 22:23:54 -07:00
Aliaksandr Valialkin	5b94246d92	docs: add Roblox case study	2023-07-06 22:23:13 -07:00
Aliaksandr Valialkin	ff96e9cfc7	docs/Single-server-VictoriaMetrics.md: fix link to Storage section after the `ab2d184e42`	2023-07-06 22:22:42 -07:00
Aliaksandr Valialkin	5a12a518a3	lib/logstorage: fix `make test-pure` tests	2023-07-06 22:22:08 -07:00
Aliaksandr Valialkin	f2f9532fa5	lib/httputils: fix test after `b49d04b3dc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459	2023-07-06 22:21:43 -07:00
Roman Khavronenko	bd5abb74fd	vmctl: interrupt explore procedure in influx mode if no numeric fields were found (#4576 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:21:18 -07:00
Denys Holius	ef29a41824	docs: use printf for Makefile:copy-docs section (#4548 ) printf handles new line char for cross-platform use	2023-07-06 22:20:53 -07:00
Roman Khavronenko	9cfd8d6b86	Docs retention (#4568 ) * docs: mention parts and partitions in Retention section Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:19:43 -07:00
Roman Khavronenko	41f0ed48eb	docs: follow-up after `9da638aa66` (#4572 ) `9da638aa66` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:18:54 -07:00
Sergey	747c39d714	security: update base Alpine image to 3.18.2 to avoid security risks (#4571 ) libcrypto3 and libssl3 in Alpine 3.18.0 have versions `3.1.0-r4` which contains CVE-2023-2650: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2650 Use ALpine image 3.18.2 which contains fixed versions of libssl3 and libcrypto3: 3.1.1-r0 NB: In Openshift these containers are marked as vulnerabilities because of these CVEs.	2023-07-06 22:12:20 -07:00
Dmytro Kozlov	dd412a3757	app/vmalert: show on UI groups error after reload config (#4543 ) show on UI groups error after reload config https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4076 Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:11:36 -07:00
Roman Khavronenko	7d1e80e6e2	bump to 1.91.3 (#4565 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:10:54 -07:00
Zakhar Bessarab	e42f856b56	app/vmagent/remotewrite: fix error message for auth config (#4545 ) Error message will be present for any auth error, but message claims an error is about OAuth2 configuration which is confusing. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 22:10:13 -07:00
Haleygo	b029286298	fix parse for invalid partial RFC3339 format (#4539 ) The validation was needed for covering corner cases when storage is tested with data from 1970. This resulted into unexpected search results, as year was parsed incorrectly from the given timestamp. Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 22:09:35 -07:00
Artem Navoiev	7ff0ac1a33	docs: add command-line flags (#4550 ) * add command-line flags Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 22:08:21 -07:00
Nikolay	68879061be	docs: adds v1.91.3 release docs (#4561 )	2023-07-06 22:06:58 -07:00
Max Golionko	75d9914916	CI: disable docker scan, enable auto release to sandbox (#4476 ) * disable docker scan * disable nightly, enable auto release to sandbox * remove whitespace	2023-07-06 22:02:48 -07:00
Zakhar Bessarab	b801360075	docs: clarify downsampling periods requirements (#4542 ) It is required for periods to be multiplies, but it was not stated clearly in documentation. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:58:40 -07:00
Yury Molodov	8c190ec8fb	vmui: fix app routing issues (#4408 ) The change focuses on rectifying inconsistencies in the navigation behavior of the application and eliminating issues encountered when manually altering the URL. The key updates include: - Refactoring of the routing mechanism to handle all possible routes and their states. - Enhancement of the React Router usage to ensure a smoother navigation experience. - Handling application state when the URL is manually changed.	2023-07-06 21:58:09 -07:00
Zakhar Bessarab	7eeb2d553f	docs/operator: add note about selectors for VMProbe (#4541 ) Added a line for `probeSelector` and links to objects selectors section to make it easier to find more details about selectors. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:57:21 -07:00
Alexander Marshalov	677c8a5465	show backup progress percentage in vmbackup log during backup uploading and restoring progress percentage in vmrestore log during backup downloading (#4460 ) (#4530 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-06 21:56:54 -07:00
Roman Khavronenko	cf433c066a	vmauth: expose latency metrics per user (#4525 ) expose `vmauth_user_request_duration_seconds` and `vmauth_unauthorized_user_request_duration_seconds` summary metrics for measuring requests latency per user. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:55:37 -07:00
Roman Khavronenko	8a15397b5c	vmauth: rm ip filters from non-ent config example (#4526 ) It is impossible to run OS vmauth with the provided config. The example of using ip filters should be only a part of docs. All other examples should work seamlessly with OS version. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:52:51 -07:00
Haleygo	9e49a9e924	vmalert: add `vmalert_remotewrite_sent_duration_seconds_total` metric (#4517 ) add `vmalert_remotewrite_sent_duration_seconds_total` metric	2023-07-06 21:51:31 -07:00
Dmytro Kozlov	316fe82dd6	docs: scroll to center of the view port (#4515 )	2023-07-06 21:50:31 -07:00
Craig Rodrigues	4e0ca99da9	docs/VictoriaLogs: Fix curl command for fetching binary	2023-07-06 21:50:05 -07:00
Dmytro Kozlov	7498d139ca	docs: add scroll to the selected element (#4508 ) * docs: add scroll to the selected element * docs: scroll to root if element not found * docs: simplify code * docs: code cleanup * docs: fix comments (fix code formatting, check element only inside sidebar container)	2023-07-06 21:48:53 -07:00
Zakhar Bessarab	175eb38695	deployment/logs-benchmark: remove unnecessary multiplication (#4502 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:48:00 -07:00
Artem Navoiev	258575441e	revert version of golanci Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:47:35 -07:00
Artem Navoiev	28301d1b41	remove deleted repo from the docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:47:07 -07:00
Roman Khavronenko	a677509b38	vmalert: make linter happy (#4509 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:46:22 -07:00
Roman Khavronenko	d5e7ea5ef3	vmalert: update retry policy for pushing data to `-remoteWrite.url` (#4504 ) By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed `-remoteWrite.retryMaxTime` (default is 30s). When retry time is exceeded vmalert drops the data dedicated for `-remoteWrite.url`. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See `-remoteWrite.retryMinInterval` and `-remoteWrite.retryMaxTime` cmd-line flags. Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-07-06 21:44:18 -07:00
Roman Khavronenko	311a81c7b0	vmalert: properly interrupt remotewrite retries on shutdown (#4505 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:43:04 -07:00
Zakhar Bessarab	dae0b428fd	deployment/logs-benchmark: update screenshot after #4500 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:42:28 -07:00
Zakhar Bessarab	b308088b3c	deployment/logs-benchmark: update dashboard (#4500 ) - remove second panel for disk usage. It is not very useful for users and brings more confusion than profit from having it. - update CPU graph to show number of used CPUs to make it less ambiguous Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:42:08 -07:00
Zakhar Bessarab	7a000159d8	docs/changelog: followup for `830dac177f` (#4499 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:41:36 -07:00
Artem Navoiev	d48e697bbd	fix jsonline endpoint in docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:40:59 -07:00
Artem Navoiev	9e9506af31	fix link vmui links in docker examples after the path chnage for vmui Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:40:16 -07:00
Aliaksandr Valialkin	b215e6e044	docs/VictoriaLogs/QuickStart.md: add an option to run VictoriaLogs from pre-built binaries	2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin	166456f340	docs/VictoriaLogs/README.md: add `Upgrading` section	2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin	881dc29ef8	docs/VictoriaLogs/README.md: typo fix	2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin	a3956b3d9c	all: update VictoriaLogs docker tag to v0.1.0-victorialogs	2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin	a9eb2409ea	app/vlstorage: export vl_active_merges and vl_merges_total metrics	2023-07-06 21:38:09 -07:00
Aliaksandr Valialkin	352429486a	Revert "app/vlselect/logsql: use buffered writer in order to save syscalls when sending big amounts of data to clients" This reverts commit `c19048dc13`. Reason for revert: it has been appeared that the net/http.ResponseWriter is already buffered, so there in no need in double bufferring	2023-07-06 21:37:38 -07:00
Aliaksandr Valialkin	19870d42c5	app/vlselect/logsql: use buffered writer in order to save syscalls when sending big amounts of data to clients	2023-07-06 21:37:38 -07:00
Aliaksandr Valialkin	33625610c6	app/vmui/Makefile: consistently use tabs instead of spaces in multi-line Makefile rules	2023-07-06 21:37:38 -07:00
Aliaksandr Valialkin	4b10432435	app/vlselect: handle vmui at /select/vmui path instead of /vmui This simplifies routing at auth proxies such as vmauth to vlselect component, which serves VMUI - just route all the requests, which start with /select/, to vlselect.	2023-07-06 21:36:28 -07:00
Aliaksandr Valialkin	33e4d51636	docs/VictoriaLogs/data-ingestion/README.md: remove trailing spaces	2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin	08634ae612	app/vlinsert/jsonline: code prettifying	2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin	64b8aa108b	docs/VictoriaLogs/README.md: small fixes	2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin	7206f91201	docs/VictoriaLogs/data-ingestion: small fixes	2023-07-06 21:35:22 -07:00
Aliaksandr Valialkin	ee720d1bbc	docs/VictoriaLogs/querying: rename VMUI -> 'Web UI' in order to reduce confusion with VictoriaMetrics UI	2023-07-06 21:35:03 -07:00
Aliaksandr Valialkin	2b43e5be80	docs/VictoriaLogs: small fixes	2023-07-06 21:34:41 -07:00
Artem Navoiev	eaa74595dc	add information about how to access web UI Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:34:16 -07:00
Alexander Marshalov	c6dba396db	added docs for vmui in victorialogs (#4494 )	2023-07-06 21:33:49 -07:00
Aliaksandr Valialkin	772852ff4f	app/vlselect/logsql: properly handle the error from ParseLogMessage	2023-07-06 21:33:22 -07:00
Dmytro Kozlov	caf4743e45	app/victoria-logs: remove header control (#4493 )	2023-07-06 21:33:00 -07:00
Alexander Marshalov	db910dd336	removed debug message from jsonlines handler of victorialogs (#4492 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-06 21:32:38 -07:00
dmitryk-dk	058fbbdb16	app/victoria-logs: add vmui dependecies	2023-07-06 21:32:19 -07:00
Alexander Marshalov	c12b5250c7	added more info and examples about data ingestion and collectors to VictoriaLogs docs (#4490 )	2023-07-06 21:31:56 -07:00
Yury Molodov	a04a206cd2	vmui: logs explorer (#4484 ) * feat: add a logs page * app/vixtoria-logs: add handlers for vmui * feat: add group logs * feat: add logs build * app/vixtoria-logs: update make file * app/vixtoria-logs: cleanup make * app/vixtoria-logs: fix description * fix: correct url for logs * fix: save display view in query params * fix: change logo for logs build * app/vixtoria-logs: remove dashboards from vlselect * app/vixtoria-logs: enable user --------- Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>	2023-07-06 21:31:33 -07:00
Artem Navoiev	97505895e3	fix linter for logs benchmark Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:31:03 -07:00
Alexander Marshalov	d9d759bc90	jsonline support for data ingestion in vlinsert (#4487 ) added json lines / json stream format for ingestion to vlinsert	2023-07-06 21:30:35 -07:00
Artem Navoiev	b16a5ee705	use latest image in example for VictoriaLogs and VictoriaMetrics to simplify the flow Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:30:15 -07:00
Artem Navoiev	a972567faa	docs: change wording in victorialogs benchmarks section Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:29:55 -07:00
Artem Navoiev	b33f7a3181	docs: change wording in victorialogs benchmarks section Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:29:34 -07:00
Artem Navoiev	d9b625719f	fix typos Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:29:11 -07:00
Zakhar Bessarab	7ab9a4d5dc	docs/VictoriaLogs: add benchmarks section into readme Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:28:48 -07:00
Zakhar Bessarab	ed18c503ac	docs/VictoriaLogs: add reference to benchmark setup in readme Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:28:27 -07:00
hagen1778	b43d27275d	docs/metricsql: fix typo in expression Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:28:01 -07:00
Alexander Marshalov	05ac9618c0	improved vector example for victoria logs Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-06 21:27:23 -07:00
Artem Navoiev	265dd66191	remove grafana notion from vector readme Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:26:55 -07:00
Artem Navoiev	d058f24650	examples add README Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:26:33 -07:00
Artem Navoiev	7c588af66e	remove docker from filebeat syslog Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:26:00 -07:00
Artem Navoiev	82b49d7194	Add docker compose examples: filebeat(docker, syslog), fluentbit(docker), logstash, vector(docker) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:25:31 -07:00
Aliaksandr Valialkin	efee71986f	app/vlselect/logsql: sort query results by _time if their summary size doesnt exceed -select.maxSortBufferSize	2023-07-06 21:25:00 -07:00
Roman Khavronenko	4e99bf8c9e	docs/vmalert: specify version requirements for new features (#4480 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 21:24:38 -07:00
Zakhar Bessarab	429324a432	deployment/logs-benchmark: add benchmark setup (#4470 ) * deployment/logs-benchmark: add benchmark setup Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * deployment/logs-benchmark: update readme info - add sizes of uncompressed logs - add more info about dashboard Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-07-06 21:24:08 -07:00
Aliaksandr Valialkin	c77f729680	docs/VictoriaLogs/README.md: typo fix	2023-07-06 21:23:45 -07:00
Aliaksandr Valialkin	74ac4722a9	docs/VictoriaLogs/README.md: add missing link to quick start docs	2023-07-06 21:23:21 -07:00
Aliaksandr Valialkin	fd6c2dd02e	docs/VictoriaLogs: change the structure of the docs in order to be more maintainable The change is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4477	2023-07-06 21:22:59 -07:00
Aliaksandr Valialkin	e21b3bceab	app/vlinsert/elasticsearch: allow empty lines in Elasticsearch bulk protocol Empty lines may appear there during debugging and custom client implementation	2023-07-06 21:22:22 -07:00
Aliaksandr Valialkin	7f146543ef	app/vlinsert/elasticsearch: optimize parsing command line Just search for "create" or "index" substrings there instead of spending CPU time on its parsing	2023-07-06 21:21:57 -07:00
Aliaksandr Valialkin	ca41a164cc	docs/url-examples.md: add missing `Content-Type: application/json` header in curl examples for json data ingestion If the `Content-Type: application/json` request header isn't set, then the server can improperly consume the request body when parsing request parameters	2023-07-06 21:21:34 -07:00
Aliaksandr Valialkin	c8cffc9b1b	app/vlstorage: log -storageDataPath and basic stats for the opened storage	2023-07-06 21:21:00 -07:00
Aliaksandr Valialkin	0edfa06daa	docs/Single-server-VictoriaMetrics.md: update link to JSON stream format, since the previous link became broken	2023-07-06 21:20:32 -07:00
Aliaksandr Valialkin	1c39af56ab	app/victoria-logs: add ability to debug data ingestion by passing `debug` query arg to data ingestion API	2023-07-06 21:19:58 -07:00
Aliaksandr Valialkin	5fdf82657f	docs/VictoriaLogs: mention that VictoriaLogs supports multitenancy and out of order logs ingestion	2023-07-06 21:19:33 -07:00
Aliaksandr Valialkin	61f9b25782	docs/Single-server-VictoriaMetrics.md: refer to Slack chat at the top of the docs	2023-07-06 21:18:59 -07:00
Artem Navoiev	28bef227c8	change title of the operator doc to match title frontmatter Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-07-06 21:18:21 -07:00
Roman Khavronenko	d4ee505f6f	vmalert: retry all errors except 4XX status codes (#4461 ) vmalert: retry all errors except 4XX status codes Retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 17:34:32 -07:00
Yury Molodov	0ad966a898	vmui: memory leak fix (#4455 ) * fix: optimize the preparation of data for the graph * fix: optimize tooltip rendering * fix: optimize re-rendering of the chart * vmui: memory leak fix	2023-07-06 17:33:54 -07:00
Aliaksandr Valialkin	8b479e9122	docs/VictoriaLogs/LogsQL.md: typo fixes	2023-07-06 17:33:18 -07:00
Aliaksandr Valialkin	f4137a28ff	docs/VictoriaLogs/LogsQL.md: typo fixes	2023-07-06 17:33:18 -07:00
Aliaksandr Valialkin	cbfbaf37f3	docs/VictoriaLogs/LogsQL.md: typo fix	2023-07-06 17:32:51 -07:00
Aliaksandr Valialkin	c96cdc75c2	docs/VictoriaLogs/LogsQL.md: typo fix	2023-07-06 17:32:27 -07:00
Aliaksandr Valialkin	4d1c2b1722	docs/VictoriaLogs/LogsQL.md: typo fixes	2023-07-06 17:32:06 -07:00
Aliaksandr Valialkin	b66fbf8509	docs/VictoriaLogs/README.md: typo fixes	2023-07-06 17:31:45 -07:00
Aliaksandr Valialkin	137cbc5529	docs/VictoriaLogs/README.md: document how to run docker image for VictoriaLogs	2023-07-06 17:31:14 -07:00
Aliaksandr Valialkin	2db55cd081	app/vmctl/utils: properly use timezone in TestGetTime()	2023-07-06 17:30:51 -07:00
Aliaksandr Valialkin	374890294e	app/victoria-logs: initial code release	2023-07-06 17:30:05 -07:00
Aliaksandr Valialkin	de574e7128	lib/storage: do not create flock.lock files at partition directories, since it is created at the Storage level	2023-07-06 17:26:37 -07:00
Aliaksandr Valialkin	833a0e25a7	lib/netutil: ignore arificial timeout generated by net/http.Server This prevents from the inflated vm_tcplistener_read_timeouts_total counter	2023-07-06 17:26:15 -07:00
Aliaksandr Valialkin	115667df82	lib/mergeset: do not create flock.lock file at mergeset table, since it is created at the lib/storage.Storage level	2023-07-06 17:25:45 -07:00
Aliaksandr Valialkin	ed5f4a0c5a	lib/fs: add ReaderAt.Path() function This function is going to be used in VictoriaLogs	2023-07-06 17:25:19 -07:00
Aliaksandr Valialkin	4c80193a86	lib/encoding: add MarshalBool/UnmarshalBool and GetUint32s/PutUint32s functions These functions are going to be used by VictoriaLogs	2023-07-06 17:24:52 -07:00
Aliaksandr Valialkin	d01f0a89db	lib/cgroup: add SetGOGC() function This function is going to be used by VictoriaLogs	2023-07-06 17:24:31 -07:00
Aliaksandr Valialkin	af6c14d5e7	lib/bytesutil: substitute parentheses with slashes in ByteBuffer.Path() output, so it can be passed to path manipulating functions This is needed for the upcoming VictoriaLogs	2023-07-06 17:23:52 -07:00
Aliaksandr Valialkin	427ce69426	app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils While at it, move app/vmselect/bufferedwriter to lib/bufferedwriter, since it is going to be used in VictoriaLogs	2023-07-06 17:22:23 -07:00
Aliaksandr Valialkin	46210c4d5e	lib/promutils.ParseTime(): add support for timestamps in milliseconds See https://stackoverflow.com/questions/76437098/how-to-handle-time-unit-and-step-while-ingesting-or-querying-in-victoriametrics/76438405 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459	2023-07-06 17:11:54 -07:00
Nikolay	dd7ebd6779	lib/storage: creates parts.json on start-up if it not exists. (#4450 ) * lib/storage: creates parts.json on start-up if it not exists. It fixes migrations from versions below v1.90.0. Previously parts.json was created only after successful merge. But if merge was interruped for some reason (OOM or shutdown), parts.json wasn't created and partitions left after interruped merge weren't properly deleted. Since VM cannot check if it must be removed or not. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update lib/storage/partition.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-07-06 17:10:26 -07:00
Roman Khavronenko	09c05608f2	lib/storage: add comment for how `mustBeDeleted` field should be used (#4454 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 17:02:44 -07:00
Roman Khavronenko	3937da289a	docs/ReleaseGuide: mention auth for Docker (#4453 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 17:02:07 -07:00
Roman Khavronenko	0e3e045f5b	docs: mention errors processing for /api/v1/import API (#4448 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4446 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 17:01:34 -07:00
Roman Khavronenko	897d17a5b3	lib/mergeset: add comment for how `mustBeDeleted` field should be used (#4449 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 17:00:55 -07:00
Dmytro Kozlov	b32a270da7	vmctl: increase retry backoff policy delay (#4447 ) vmctl: update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases	2023-07-06 17:00:06 -07:00
Roman Khavronenko	b76c0d182c	docs/vmalert: mention `same labelset` error in docs (#4443 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:56:42 -07:00
Dmytro Kozlov	2e81c5f740	vmctl: finish retries if context canceled (#4442 ) vmctl: interrupt backoff retries if import context is cancelled Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-07-06 16:56:00 -07:00
Roman Khavronenko	0e0b7bf87f	docs/release-guide: update instructions (#4391 ) docs/release-guide: update instructions for MacOS users Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:54:46 -07:00
Alexander Marshalov	4084dba9e4	fixed service name detection for consulagent service discovery in case of a difference in service name and service id (#4390 ) (#4439 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-07-06 16:53:29 -07:00
Roman Khavronenko	ecd7ec4832	Dashboard upd (#4438 ) dashboards: update dashboard for single-node version * add anonymous mem usage panel; * add syscall rate panel; * add location to logs panel; * update legend for panels to reflect instance name; * update queries to aggregate per instance. dashboards: update dashboard for cluster version * add syscall rate panel; * add drilldown to logs panel. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:49:42 -07:00
greynix	82a2d6aba5	docs/oprator/api.md: corrected broken links to Kubernetes documentation (#4433 )	2023-07-06 16:47:44 -07:00
Roman Khavronenko	2e3c3cf7ea	docs: make docs-sync (#4430 ) docs: fix typo Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:47:10 -07:00
Roman Khavronenko	91612b38cd	docs: mention stream aggregation as more efficient approach for aggregation (#4429 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:46:17 -07:00
Roman Khavronenko	5518cb2f9a	docs/keyConcepts: explicitly specify time unit for API args (#4428 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-06 16:45:27 -07:00
Aliaksandr Valialkin	3bc3fb6adf	lib/vmselectapi: move the code for checking the expected client errors into a isExpectedError() function	2023-07-06 16:37:59 -07:00
Aliaksandr Valialkin	ed868f47f9	docs/CHANGELOG.md: remove the change regarding http2 support at vmagent This is a follow-up for `8a07621a0c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283	2023-07-06 16:06:44 -07:00
Aliaksandr Valialkin	5b8095a30a	lib/promscrape: disable support for service discovery and metrics scrape via http2 Reasons for disabling http2: - http2 is used very rarely comparing to http for Prometheus metrics exposition and service discovery - http2 is much harder to debug than http - http2 has very bad security record because of its complexity - see https://portswigger.net/research/http2 VictoriaMetrics components are compiled with nethttpomithttp2 tag because of these issues. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4274 This is a follow-up for `72c3cd47eb`	2023-07-06 16:04:31 -07:00
Aliaksandr Valialkin	dff199a745	app/vmselect/graphite: follow-up after `c7884f8686` - Consistently use -search.maxGraphiteTagValues for limiting tag values from auto-complete API - Use -search.maxGraphiteSeries for limiting paths (aka series), which can be returned from Graphite series API - Clarify the change in docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841	2023-07-06 15:19:07 -07:00
Aliaksandr Valialkin	4f6eb775df	Makefile: remove trailing whitespace from copy-docs comments	2023-07-06 11:01:39 -07:00
Aliaksandr Valialkin	6a3cee5c2c	lib/promscrape/discoveryutils: re-use checkRedirect function for both client and blockingClient Also document follow_redirects option at https://docs.victoriametrics.com/sd_configs.html#http-api-client-options This is a follow-up for `b3d0ff463a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282	2023-07-06 10:52:13 -07:00
Aliaksandr Valialkin	eb47ad4b69	app/vmselect/netstorage: remove runtime.Gosched() call from unpackWorker() This should improve scalability of unpackWorker() on systems with many CPU cores. This is a follow-up for `a2ecf4fa4a` and `16f3b279a2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966	2023-07-06 10:07:42 -07:00
Alexander Marshalov	b3f8bb5b50	vmbackupmanager bugfixes: (#577 ) - error on running with empty -dst dir and without -runOnStart - error on restoring with backup, created before v1.90.0	2023-07-05 22:08:04 -07:00
Aliaksandr Valialkin	ec75d9097d	app/vmselect/netstorage: follow-up after `11ac551d52` - Clarify the scope of the fix at docs/CHANGELOG.md - Handle the case when -search.maxSamplesPerSeries limit is exceeded in the same way as the -search.maxSamplesPerQuery limit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472	2023-07-05 21:13:34 -07:00
Aliaksandr Valialkin	643e99a157	app/vmselect/netstorage: improve code readability a bit after `6c84b61893` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4364	2023-07-05 20:48:38 -07:00
Roman Khavronenko	11ac551d52	app/vmselect/netstorage: properly process `-search.maxSamplesPerQuery` limit (#4472 ) Properly return the error to user when `-search.maxSamplesPerQuery` limit is exceeded. Before, user could have received a partial response instead. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-23 13:17:34 +02:00
Zakhar Bessarab	bf4120a3d9	lib/vmselectapi: extend error handling to ignore "reset by peer" (#4498 ) This is a followup for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418 to also handle "connection reset by peer" errors in connection handling logic. This error can be triggered just the same as described in original PR: when query was closed on vmselect side and connection has been interrupted. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-06-22 11:24:18 +02:00
Dmytro Kozlov	88ac6116bd	docs: clarify `-retentionPeriod` flag usage (#4417 ) app/vmstorage: clarify the min value for `-retentionPeriod` flag Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> (cherry picked from commit `24f34347f1`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:44:22 +02:00
Roman Khavronenko	4624fda00d	all: update Go builder from Go1.20.4 to Go1.20.5 (#4427 ) See https://github.com/golang/go/issues?q=milestone%3AGo1.20.5+label%3ACherryPickApproved Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `476c7bdd6f`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:42:15 +02:00
hagen1778	dde01c826d	lib/vmselectapi: properly check for net.ErrClosed This error may be wrapped in another error, and should normally be tested using `errors.Is(err, net.ErrClosed)`. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:42:03 +02:00
Roman Khavronenko	d677c2a5a6	lib/promscrape/discoveryutils: properly check for net.ErrClosed (#4426 ) This error may be wrapped in another error, and should normally be tested using `errors.Is(err, net.ErrClosed)`. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `dfe53a36fc`)	2023-06-09 10:41:07 +02:00
Roman Khavronenko	c42365dc31	docs/changelog: mention `a6a7795b9e` change (#4425 ) docs/changelog: mention `a6a7795b9e` change `a6a7795b9e` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `d4c314d628`)	2023-06-09 10:41:07 +02:00
Zakhar Bessarab	bcece4c5ce	doc: changelog followup for #4420 fix (#4421 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `9a490d0b5c`)	2023-06-09 10:41:07 +02:00
Zakhar Bessarab	7925e9698f	app/vmagent/remotewrite: fix vmagent panic on shutdown (#4407 ) app/vmagent/remotewrite: fix vmagent panic on shutdown Currently, when vmagent is stopping it first flushes pending series in remote write context and proceeds to stop streaming aggregation. This leads to streaming aggregation being unable to write results into pending timeseries (since it is already nil) and panic. This can lead to losing some aggregation results being lost almost silently. The fix is reordering flow to first stop streaming aggregation and flush all pending time series after that. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> (cherry picked from commit `ce7141383d`)	2023-06-09 10:40:52 +02:00
Roman Khavronenko	3544e66a95	docs: mention checksums and data corruption (#4404 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `96b40b044c`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:40:48 +02:00
Roman Khavronenko	71e9eaae0a	docs/keyConcepts: replace `lookback window` with more clear desc (#4405 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `bfa0fd3eb3`)	2023-06-09 10:40:26 +02:00
Roman Khavronenko	fb9b8f6b1b	app/vmagent: mention `enable_http2` in changelog (#4403 ) Follow-up after `72c3cd47eb` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `3305a6901c`)	2023-06-09 10:40:24 +02:00
Haleygo	6edf94c4b9	vmagent:scrape config support enable_http2 (#4295 ) app/vmagent: support `enable_http2` in scrape config This change adds HTTP2 support for scrape config and improves compatibility with Prometheus config. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283 (cherry picked from commit `72c3cd47eb`)	2023-06-09 10:40:17 +02:00
Roman Khavronenko	d9131d71cd	docs/CHANGELOG.md: cut v1.91.2 (#4393 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `cc739e3f8d`)	2023-06-09 10:40:13 +02:00
Roman Khavronenko	c9f6b34a82	Bump 1.91.2 (#4394 ) * deployment/docker: update VictoriaMetrics version from v1.91.1 to v1.91.2 in docker compose files Signed-off-by: hagen1778 <roman@victoriametrics.com> * deployment/marketplace: update VictoriaMetrics version from v1.91.1 to v1.91.2 in marketplace files Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2d6a3bc61f`)	2023-06-09 10:40:10 +02:00
Dmytro Kozlov	dd89fb2e12	app/vmctl: add verbose output for docker installations or when TTY isn't available (#4333 ) * app/vmctl: add verbose output for docker installations or when TTY isn't available * app/vmctl: fix tests * app/vmctl: make vmctl interactive if no tty * app/vmctl: cleanup * app/vmctl: add comment --------- Co-authored-by: Nikolay <nik@victoriametrics.com> (cherry picked from commit `fc5292d8ed`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:39:38 +02:00
Dmytro Kozlov	c5debee3f4	app/{graphite,netstorage,prometheus}: fix graphite search tags api limits, remove redudant limit from SeriesHandler handler (#4352 ) * app/{graphite,netstorage,prometheus}: fix graphite search tags api limits, remove unused limit from SeriesHandler handler, * app/{graphite,netstorage,prometheus}: use search.maxTagValues for Graphite * app/{graphite,netstorage,prometheus}: update CHANGELOG.md * app/{graphite,netstorage,prometheus}: use own flags for Graphite API * app/{graphite,netstorage,prometheus}: cleanup * app/{graphite,netstorage,prometheus}: cleanup * app/{graphite,netstorage,prometheus}: update docs --------- Co-authored-by: Nikolay <nik@victoriametrics.com> (cherry picked from commit `c7884f8686`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-09 10:39:12 +02:00
Roman Khavronenko	dfb05c884b	lib/vmselectapi: suppress "broken pipe" error logs on vmstorage side (#4418 ) The "broken pipe" error is emitted when the connection has been interrupted abruptly. It could happen due to unexpected network glitch or because connection was interrupted by remote client. In both cases, remote client will notice connection breach and handle it on its own. No need in logging this error on both: server and client side. This change should reduce the amount of log noise on vmstorage side. In the same time, it is not expected to lose any information, since important logs should be still emitted by the vmselect. To conduct an experiment for testing this change see the following instructions: 1. Setup vmcluster with at least 2 storage nodes, 1 vminsert and 1 vmselect 2. Run vmselect with complexity limit checked on the client side: `-search.maxSamplesPerQuery=1` 3. Ingest some data and query it back: `count({__name__!=""})` 4. Observe the logs on vmselect and vmstorage side Before the change, vmselect will log message about complexity limits exceeded. When this happens, vmselect closes network connections to vmstorage nodes signalizing that it doesn't expect any data back. Both vmstorage processes will try to push data to the connection and will fail with "broken pipe" error, means that vmselect closed the connection. After the change, vmstorages should remain silent. And vmselect will continue emittin the error message about complexity limits exceeded. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-08 08:31:05 -07:00
Roman Khavronenko	a1b6a9317d	vmalert: fix nil map assignment (#4392 ) * vmalert: fix nil map assignment The storage instance with nil map params was created for remote-read purposes. And before change `7a9ae9de0d` this map was ignored in ApplyParams. Now, it started to be used and vmalert panics in runtime. The fix properly inits map for at `NewVMStorage` and verifies it is not nil on assignment in `ApplyParams`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: add to changelog Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: properly clone Storage params Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: properly clone Storage params Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: properly clone Storage params Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `de94812088`)	2023-06-02 13:29:51 +02:00
Roman Khavronenko	b00509704f	deployment/docker: update VictoriaMetrics version from v1.91.0 to v1.91.1 in docker compose files (#4387 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `2c664a6d12`)	2023-06-02 13:29:50 +02:00
Roman Khavronenko	e0d565bce1	docs/CHANGELOG.md: cut v1.91.1 (#4386 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `b771152039`)	2023-06-02 13:29:49 +02:00
Nikolay	89c036f4f6	docs/changlelog: mention `6c84b61` (#4384 ) (cherry picked from commit `2c876227e4`)	2023-06-02 13:29:48 +02:00
Roman Khavronenko	fde5a59726	app/vmalert: follow-up after `7a9ae9de0d` (#4381 ) `7a9ae9de0d` Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `eccecdf177`)	2023-06-02 13:29:47 +02:00
Roman Khavronenko	598671c0e8	docs: mention fix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 (#4382 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `4b5faf7efb`)	2023-06-02 13:29:46 +02:00
Dmytro Kozlov	ce98e68fe1	app/vmui: fix behavior when changing url in global settings (#4332 ) * app/vmui: fix behavior when changing url in global settings * app/vmctl: minor fix * app/vmui: fix behavior when changing url in global settings (cherry picked from commit `9843ec0e1d`)	2023-06-02 13:29:46 +02:00
Nikolay	5066907317	docs: mention recent changes at changelog (#4379 ) (cherry picked from commit `a0bf8f233f`)	2023-06-02 13:29:45 +02:00
Roman Khavronenko	7da0c37d2b	docs: clarify deduplication docs (#4371 ) The purpose of the change is too highlight what HA pair is and how deduplication needs identical labels to be present in raw samples. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4367 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `8185c2466c`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-02 13:29:45 +02:00
gsakun	72b72f06a0	app/vmalert: fix datasource.roundDigits Parameter (#4341 ) app/vmalert: fix querybuild clone and extraParams merge logic See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340 (cherry picked from commit `20dc3db71e`)	2023-06-02 13:29:16 +02:00
Denys Holius	f78e34c0b0	docs/Quick-Start.md: adds missed command to 'Starting VM-Cluster via Docker' section (#4375 ) (cherry picked from commit `0bca7d49b9`)	2023-06-02 13:29:15 +02:00
Nikolay	043431093a	app/vmauth: properly handle LOCAL proxy protocol command (#4373 ) app/vmauth: properly handle LOCAL proxy protocol command It is required for handling health checks from load balancers https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335 (cherry picked from commit `f263031fe9`)	2023-06-02 13:29:15 +02:00
Roman Khavronenko	c862952cb0	docs: mention disk space requirements for downsampling (#4369 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `a0c040ea58`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-02 13:29:14 +02:00
Roman Khavronenko	288e8fec6b	vmalert: mention default value for `external.url` flag (#4365 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1b24f4729b`)	2023-06-02 13:28:08 +02:00
Artem Navoiev	c9c4c92e6e	docs: fix enterprise page title to match with official name Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `a8c58fc145`)	2023-06-02 13:28:06 +02:00
Artem Navoiev	6dca7d245b	docs: fix markdown headers ordering Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `5b0e8c797c`)	2023-06-02 13:28:05 +02:00
Artem Navoiev	27960fd6ec	docs: fix markdown for title Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `83c1944184`)	2023-06-02 13:28:04 +02:00
Artem Navoiev	bbd1dd9d03	fix image path Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `6967c3cb95`)	2023-06-02 13:28:04 +02:00
Artem Navoiev	756f0e1545	docs: change images from markdown tag to html for migration Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `8a4c89ea22`)	2023-06-02 13:28:04 +02:00
Roman Khavronenko	74c4c1d909	vmalert: properly form assets address if httpPrefix set (#4351 ) Properly form path to static assets in WEB UI if `http.pathPrefix` set. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `51cea6cad4`)	2023-06-02 13:28:03 +02:00
Roman Khavronenko	a0825ca5cd	docs: mention multi-tenancy in docs (#4357 ) The update should make understanding of multi-tenancy more clear for influxdb users. Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `1c3f50f791`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-02 13:28:02 +02:00
Artem Navoiev	8dd2a263b9	desribe old link param Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `bf4711ecba`)	2023-06-02 13:25:49 +02:00
Artem Navoiev	5e6fe50c53	return information about cluster Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `8f09569cb8`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-02 13:25:48 +02:00
Artem Navoiev	3185b8048c	update docs-sync Makefile command, add hugo front-matter Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `f791811b15`) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-02 13:24:41 +02:00
Haleygo	73a8f763a0	vmagent:support follow_redirects on SD level (#4286 ) * vmagent:support follow_redirects on SD level * fix follow_redirects on sd level https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282 (cherry picked from commit `b3d0ff463a`)	2023-06-02 13:19:35 +02:00
Nikolay	e3ce736ce2	app/vmselect/graphite: fixes tests for arm (#4348 ) at arm based CPUs only 9 digits after comma matches for tests. Especially at holtWinters functions. Since it only takes effect at tests it makes no sense for changing float prescision at actual functions (cherry picked from commit `228ea03bda`)	2023-06-02 13:19:34 +02:00
Artem Navoiev	ec97b851bb	remove absolute links, fix aliase Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `434358b551`)	2023-06-02 13:19:33 +02:00
Artem Navoiev	db3d34ca9a	fix title in multi-regional-setup-dedicated-regions.md Signed-off-by: Artem Navoiev <tenmozes@gmail.com> (cherry picked from commit `c1ad48edf9`)	2023-06-02 13:19:33 +02:00
Roman Khavronenko	033e1ac3d2	vmalert: do not return nil `rules` for /api/v1/rules (#4344 ) The fix addresses a case when vmalert is configured with a group which has `name`, but doesn't have `rules` configured. In this case it still returns a `nil` instead of `[]` slice. Fixing this via current commit. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `66ed6fe62f`)	2023-06-02 13:19:33 +02:00
Roman Khavronenko	5950a94e63	vmalert: fix the typo in popup (#4331 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit `76f7e66d8e`)	2023-06-02 13:19:32 +02:00
Roman Khavronenko	576e59d82c	cluster: standardize default HTTP responses (#4368 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-06-01 10:26:52 +02:00
Haleygo	6c84b61893	vmselect:fix init sn take too much time (#4366 ) * vmselect: descrease start time for vmselect https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4364	2023-05-30 13:04:31 +02:00
Aliaksandr Valialkin	c30f0e51d7	lib/promrelabel: use monospace font at textarea for writing relabel configs on /metric-relabel-debug and /target-relabel-debug pages This simplifies visual inspection of indentation in yaml configs	2023-05-18 20:49:47 -07:00
Aliaksandr Valialkin	934a7f485c	app/vmselect: log locations of sendPrometheusError() calls Previously the location inside the sendPrometheusError() was logged. This could make hard investigating error locations via `vm_log_messages_total` metric.	2023-05-18 20:39:50 -07:00
Aliaksandr Valialkin	f67a2f1e0f	docs/CHANGELOG.md: document v1.79.13 LTS release	2023-05-18 19:56:43 -07:00
Aliaksandr Valialkin	3c4aeddfde	docs/CHANGELOG.md: document v1.87.6 LTS release	2023-05-18 17:50:22 -07:00
Aliaksandr Valialkin	b73a2336f0	deployment/docker: update VictoriaMetrics version from v1.90.0 to v1.91.0 in docker compose files	2023-05-18 15:34:54 -07:00
Aliaksandr Valialkin	63540f3ff1	docs/Release-Guide.md: fix a link to sandbox environment The link became broken after `5f9d3f9cb5`	2023-05-18 15:34:52 -07:00
Aliaksandr Valialkin	93f934eae3	deployment/docker/Makefile: use alpine 3.17.3 instead of alpine 3.18.0 for certs image, since alpine 3.18.0 doesnt work for cross-platform builds	2023-05-18 14:11:02 -07:00
Aliaksandr Valialkin	deab51e708	docs/CHANGELOG.md: cut v1.91.0	2023-05-18 12:37:20 -07:00
Aliaksandr Valialkin	131a1146d9	vendor: `make vendor-update`	2023-05-18 12:22:16 -07:00
Aliaksandr Valialkin	1ff67bb036	app/vmselect/vmui: run `make vmui-update` after `39c1b0f8d1`	2023-05-18 12:15:22 -07:00
Aliaksandr Valialkin	0ebfb91aba	lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) This reverts the following commits: - `e0e16a2d36` - `2ce02a7fe6` The reason for revert: the updated logic breaks assumptions made when fixing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698 . For example, if a time series stop receiving new samples during the first day after the indexdb rotation, there are chances that the time series won't be registered in the new indexdb. This is OK until the next indexdb rotation, since the time series is registered in the previous indexdb, so it can be found during queries. But the time series will become invisible for search after the next indexdb rotation, while its data is still there. There is also incompletely solved issue with the increased CPU and disk IO resource usage just after the indexdb rotation. There was an attempt to fix it, but it didn't fix it in full, while introducing the issue mentioned above. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 TODO: to find out the solution, which simultaneously solves the following issues: - increased memory usage for setups high churn rate and long retention (e.g. what the reverted commit does) - increased CPU and disk IO usage during indexdb rotation ( https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 ) - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2023-05-18 11:28:54 -07:00
Dmytro Kozlov	6da5cfa3a8	app/vmctl: set default value for --vm-native-step-interval flag (#4327 ) * app/vmctl: set default value for `--vm-native-step-interval` flag * app/vmctl: update CHANGELOG.md * app/vmctl: update CHANGELOG.md, fix docs * app/vmctl: fix typo * app/vmctl: fix typo	2023-05-18 11:03:26 -07:00
Denys Holius	211f48fed6	deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2 (#4314 ) deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2	2023-05-18 10:48:35 -07:00
Aliaksandr Valialkin	0397b3f0f7	lib/handshake: do not pollute logs with `cannot read hello` messages on TCP health checks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1762	2023-05-18 10:37:59 -07:00
Aliaksandr Valialkin	b22bcb6f0a	app/vmauth: allow -auth.config without `users` section of `unauthorized_user` section is present here Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4083	2023-05-18 09:44:07 -07:00
Aliaksandr Valialkin	6aa8029f30	app/vmauth: simplify the code after `4a1d29126c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4242	2023-05-17 00:38:04 -07:00
Nikolay	da115b5170	app/vmauth: retry common network dial errors (#4280 ) with tracking request body read calls it allows us to retry POST and PUT requests	2023-05-17 00:37:39 -07:00
Nikolay	953a5df43f	app/vmauth: do not return invalid credentials (#4288 ) at http response by default https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188 based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4190 Thanks @raj-kumar-j for init implementation	2023-05-17 00:10:51 -07:00
Aliaksandr Valialkin	67beb8c856	lib/storage: follow-up after `2ce02a7fe6` - Document the change at docs/CHANGELOG.md - Clarify comments for non-trivial code touched by the commit - Improve the logic behind maybeCreateIndexes(): - Correctly create per-day indexes if the indexdb rotation is performed during the first hour or the last hour of the day by UTC. Previously there was a possibility of missing index entries on that day. - Increase the duration for creating new indexes in the current indexdb for up to 22 hours after indexdb rotation. This should reduce the increased resource usage after indexdb rotation. It is safe to postpone index creation for the current day until the last hour of the current day after indexdb rotation by UTC, since the corresponding (date, ...) entries exist in the previous indexdb. - Search for TSID by (date, MetricName) in both the current and the previous indexdb. Previously the search was performed only in the current indexdb. This could lead to excess creation of per-day indexes for the current day just after indexdb rotation. - Search for (date, metricID) entries in both the current and the previous indexdb. Previously the search was performed only in the current indexdb. This could lead to excess creation of per-day indexes for the current day just after indexdb rotation.	2023-05-16 23:31:59 -07:00
Roman Khavronenko	c3b1d9ee21	lib/storage: introduce per-day MetricName=>TSID index (#4252 ) The new index substitutes global MetricName=>TSID index used for locating TSIDs on ingestion path. For installations with high ingestion and churn rate, global MetricName=>TSID index can grow enormously making index lookups too expensive. This also results into bigger than expected cache growth for indexdb blocks. New per-day index supposed to be much smaller and more efficient. This should improve ingestion speed and reliability during re-routings in cluster. The negative outcome could be occupied disk size, since per-day index is more expensive comparing to global index. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-16 23:18:11 -07:00
Aliaksandr Valialkin	bc98ea9a8d	lib/storage: reduce the unimportant logging during Storage start / stop This should improve the visibility of potentially important logs	2023-05-16 15:32:35 -07:00
Aliaksandr Valialkin	05113bba09	lib/mergeset: remove superflouos logging when opening and closing the Table The logged messages had little useful info, while they were polluting log output during VictoriaMetrics start/stop	2023-05-16 15:32:35 -07:00
Aliaksandr Valialkin	4a5b5c5020	lib/mergeset: close and open the table before making snapshots at TestTableCreateSnapshotAt() This gives guarantees that all the in-memory data is written to disk at the snapshot time. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4272 See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4316	2023-05-16 15:32:34 -07:00
Artem Navoiev	036ab185bf	update logo width in cluster doc to 300 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-16 23:05:30 +02:00
Aliaksandr Valialkin	f09745f613	lib/{mergeset,storage}: make it clear that DebugFlush() doesn't store all the recently ingested data to disk DebugFlush() makes sure that the recently ingested data becomes visible to search. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4272	2023-05-16 11:55:58 -07:00
Aliaksandr Valialkin	9af04e4140	vendor: update github.com/VictoriaMetrics/metrics from v1.23.1 to v1.24.0 This change adds process_* metrics to VictoriaMetrics components under Windows OS See https://github.com/VictoriaMetrics/metrics/pull/47	2023-05-16 11:37:18 -07:00
Aliaksandr Valialkin	4f6680f399	docs/vmbackupmanager.md: run `make docs-sync` after c7d8dda39225b716ea44df7223db5e4a125d407b	2023-05-16 11:27:39 -07:00
Alexander Marshalov	ad35081066	backup metadata are written in separate file (#560 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-16 11:24:44 -07:00
Alexander Marshalov	e895358939	added backup locking/unlocking against retention policy to vmbackupmanager (#558 ) * added backup locking/unlocking against retention policy to vmbackupmanager Signed-off-by: Alexander Marshalov <_@marshalov.org> * added docs for new commands Signed-off-by: Alexander Marshalov <_@marshalov.org> * fix review comments Signed-off-by: Alexander Marshalov <_@marshalov.org> --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-16 11:23:43 -07:00
Roman Khavronenko	42b90e5e9a	vmalert: follow-up after `669becd011` (#4318 ) * vmalert: follow-up after `669becd011` Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: follow-up after `669becd011` Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: follow-up after `669becd011` Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-16 10:14:15 -07:00
Zakhar Bessarab	a2fc912c43	lib/storage: follow-up after `a50d63c376` (#4289 ) * lib/storage: follow-up after `a50d63c376` - ensure retentionMsecs is rounded to day - remove localTimeOffset in test as localOffset is ignored when using `UnixMilli` Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/storage: restore retention timezone offset effect on retention deadline Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-16 10:13:20 -07:00
Michael Hoffmann	a99918085d	vmalert: improve retry logic for remote write (#4134 ) vmalert should not retry on 4xx status codes according to https://prometheus.io/docs/concepts/remote_write_spec/	2023-05-16 10:10:39 -07:00
Yury Molodov	5c2ed85eb9	vmui: refactor code using custom hooks (#4145 ) * refactor: replace boolean useState with useBoolean * refactor: replace useResize with useWindowSize/useElementSize * refactor: replace addEventListener with useEventListener * refactor: replace navigator.clipboard.writeText with useCopyToClipboard * fix: prevent redirect loop	2023-05-16 10:09:24 -07:00
Roman Khavronenko	afd03f87fe	vmctl: update VictoriaMetrics migration section (#4310 ) Remove unnecessary information to simplify the description and tips. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-16 10:07:47 -07:00
Artem Navoiev	b4aaeda72f	fix link in operator quick start docs.2 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-16 10:06:52 -07:00
Artem Navoiev	c771f4eccf	fix link in operator quick start docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-16 10:05:36 -07:00
Nikolay	5514b5d552	app/vminsert: allow parsing tenant id from (#4144 ) VictoriaMetrics_ProjectID and VictoriaMetrics_AccountID labels. It should help to migrate for new labels vm_account_id vm_project_id without service downtime	2023-05-16 08:16:37 -07:00
Aliaksandr Valialkin	9461d3fdfa	lib/promutils: add ParseTimeAt() function	2023-05-13 20:12:55 -07:00
Aliaksandr Valialkin	e6648d14b2	deployment/docker: update base docker image from 3.17.3 to 3.18.0 See https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html	2023-05-12 17:31:35 -07:00
Aliaksandr Valialkin	3b1e40d73f	lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString() Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284 See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4304	2023-05-12 17:02:06 -07:00
Aliaksandr Valialkin	b24da0f901	Revert "lib/promrelabel: show error message if labels not in prometheus exposition format (#4304 )" This reverts commit `193a9c3328`. Reason for revert: the commit doesn't fix the real issue with promutils.NewLabelsFromString() function, which must return error when improperly formatted Prometheus metric with labels is passed to it. See https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-example E.g. the promutils.NewLabelsFromString() must return error when the following strings are passed to it: - `{foo:"bar"}`, since `:` is disallowed in Prometheus text exposition format. The corect value is `{foo="bar"}` - `{"foo":"bar"}`, since label name shouldn't be quoted. The correct value is `{foo="bar"}`. The reverted commit introduces another set of bugs, which happily accept the following invalid input: - `{foo=~"bar"}` - `{foo!="bar"}` - `{foo!~"bar"}` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284 See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4304	2023-05-12 17:01:23 -07:00
Aliaksandr Valialkin	4df7573858	lib/protoparser/csvimport: properly parse the last empty column in CSV line Do not ignore the last empty column in CSV line. While at it, properly parse CSV columns in single quotes, e.g. `'foo,bar',baz` is parsed as two columns - `foo,bar` and `baz` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048 See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298	2023-05-12 16:59:50 -07:00
Aliaksandr Valialkin	6fd39e2000	Revert "lib/protoparser: fix skip csv line when metric can be collect from the line (#4298 )" This reverts commit `410ae99c2e`. Reason for revert: the commit masks the real issue instead of fixing it. The real issue is that the scanner.NextColumn() skips the last column if it is empty. The commit also introduces two bugs: - a panic if all the metric values in CSV line are empty - silent import of CSV lines with too small number of columns Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048 See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298	2023-05-12 16:59:11 -07:00
Yury Molodov	ff39df74d3	vmui: add notification for non-matching queries (#4301 ) vmui: add notification for non-matching queries (#4211) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4211	2023-05-12 16:58:48 -07:00
Roman Khavronenko	ddc3f0c5c5	docs: mention link to public relabeling playground (#4306 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-12 16:57:17 -07:00
Roman Khavronenko	f613d20466	docs: update docs about VMUI pages (#4305 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-12 16:56:00 -07:00
Dmytro Kozlov	24386f68db	lib/promrelabel: show error message if labels not in prometheus exposition format (#4304 ) lib/promrelabel: show error message if labels not in prometheus exposition format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284	2023-05-12 16:53:59 -07:00
Dmytro Kozlov	714236c2d8	lib/protoparser: fix skip csv line when metric can be collect from the line (#4298 ) * lib/protoparser: fix skip csv line when metric can be collect from the line * lib/protoparser: fix comment	2023-05-12 15:53:52 -07:00
Yury Molodov	c530bc639c	vmui: enhancements to top queries page (#4299 ) * feat: improvement of the top queries page * vmui/docs: enhancements to top queries page * Apply suggestions from code review --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-05-12 15:53:05 -07:00
Alexander Marshalov	f796c5dd9e	fixed error with double slash in vmbackupmanager (#557 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-11 13:38:40 -07:00
Aliaksandr Valialkin	b4d919fbc7	docs/CHANGELOG.md: improve the description of the change at `7ea2531db0` Move the change description to the group of vmui changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4213	2023-05-11 13:36:34 -07:00
Aliaksandr Valialkin	d21a244641	lib/promutils: properly parse time strings with timezones at ParseTime()	2023-05-11 13:36:00 -07:00
Roman Khavronenko	d6691e7a03	vmalert: add hints to filter buttons (#4296 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-11 13:35:27 -07:00
Denys Holius	14529208d8	app/vmctl/vm_native.go: fixed a typo in error message	2023-05-11 13:33:44 -07:00
Yury Molodov	55f77dd353	vmui: increase font-size and fix the text display (#4273 ) vmui: change default font size to 14px for better readability vmui: fix bug with missing text on buttons in safari --------- Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-05-11 13:33:06 -07:00
Dmytro Kozlov	fc9e7b650b	app/vmui: added table where Labels with the highest number of unique values show (#4271 ) * app/vmui: added Labels with the highest number of unique values * app/vmui: cleanup * app/vmui: cleanup * app/vmui: add table description * app/vmui: fix comment, updated CHANGELOG.md * app/vmui: disable links * app/vmui: added actions to the table, it will show values for selected label with the highest number of series * app/vmui: fix comment	2023-05-11 13:31:56 -07:00
Aliaksandr Valialkin	a24e08e6de	lib/bytesutil: `go fmt` after `2ec17bed2c`	2023-05-10 20:29:15 -07:00
Aliaksandr Valialkin	e6e30667b1	docs/CHANGELOG.md: fix typo after `2caf0b05c6`	2023-05-10 13:05:33 -07:00
Aliaksandr Valialkin	b7239c2221	lib/bytesutil: add benchmarks for ToUnsafeString() and ToUnsafeBytes()	2023-05-10 13:05:33 -07:00
Artem Navoiev	6108ef1758	fix typo in changelog Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-10 13:05:33 -07:00
Roman Khavronenko	fc623e2b84	Vmalert UI updates (#4276 ) * vmalert: expand rule groups on anchor click before, anchor click was only updating the URL. To expand the group, user had to click on rule's block. Now, group will toggle automatically. * vmalert: allow filtering group in web UI The new filter allows to filter groups and rules within groups by: errors only or noMatch only. The filtering supposed to help navigating big numbers of groups/rules. Filtering is reflected in URL, so can be shared as a link. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-10 13:05:05 -07:00
Roman Khavronenko	6365d97aee	vmalert: correctly update seriesFetched metric for const exprs (#4287 ) Previously, metric `vmalert_alerting_rules_last_evaluation_series_fetched` would be set to 0 for const expressions, because const expression do not match any series. This may result into a confusion: no series were matched but response isn't empty. The change updates the logic behind metric: if no series were matched but there are samples in response - use amount of samples as number of series. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-10 13:01:03 -07:00
Dmytro Kozlov	fd35f023dd	docs: Add information about datasource plugin (#4266 ) docs: Add information about datasource plugin	2023-05-10 02:24:33 -07:00
Alexander Marshalov	d321ea91f2	fixed typos in documentation and commandline flags descriptions (#4275 )	2023-05-10 02:22:06 -07:00
Aliaksandr Valialkin	64fa95dc92	vendor: update github.com/valyala/gozstd from v1.19.1 to v1.20.1	2023-05-10 02:15:22 -07:00
Aliaksandr Valialkin	1fa5223237	vendor: `make vendor-update`	2023-05-09 23:16:43 -07:00
Aliaksandr Valialkin	15618c99af	docs/CHANGELOG.md: add a link to docs about never-firing alerts Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039	2023-05-09 23:09:37 -07:00
Aliaksandr Valialkin	d2cbb34299	docs/CHANGELOG.md: document `8f4de6fa47`	2023-05-09 23:07:22 -07:00
Roman Khavronenko	81c5739529	vminsert: properly reset labels object on aggregation (#4278 ) Without reset, labels duplicates could have been added during stream aggregation. Since `ctx.Labels` is reused during processing of many series, each series will add its labels to the context. Even if the same labels were already addeded on prev iteration. Now, we reset `ctx.Labels` on each iteration to contain so labels from different series didn't interfere. This could have cause exceeding of the limit on number of labels per pushed time series. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-09 22:59:07 -07:00
Aliaksandr Valialkin	a47b9e55ac	lib/promscrape/discovery/consulagent: substitute metaPrefix with the `__meta_consulagent_` plaintext string This simplifies future code navigation and search for the specific meta-label starting from __meta_consulagent_* prefix. For example, `grep __meta_consulagent_namespace` finds the exact place where this label is defined. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3953 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4217	2023-05-09 22:58:08 -07:00
Aliaksandr Valialkin	e4c615e777	lib/fs: move common code outside arch-specific implementations of mustRemoveDirAtomic() This is a follow-up for `73b6c23271` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-05-09 22:56:40 -07:00
Aliaksandr Valialkin	a5a57a7c2a	docs/CHANGELOG.md: group changelog lines for tip release according to VictoriaMetrics apps	2023-05-09 22:55:17 -07:00
Aliaksandr Valialkin	b20bbb8f79	docs/CHANGELOG.md: document `baf456978d` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4223	2023-05-09 22:53:36 -07:00
Aliaksandr Valialkin	dbdf029f6b	docs/managed-victoriametrics/overview.md: typo fix after `51fbf58d89`	2023-05-09 22:53:05 -07:00
Aliaksandr Valialkin	7adc1665ba	docs/CHANGELOG.md: refer to the author and the pull request of the notifier_headers feature at vmalert Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260	2023-05-09 22:52:34 -07:00
Aliaksandr Valialkin	24aa295f43	docs/CHANGELOG.md: document `03150c8973` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204	2023-05-09 22:50:10 -07:00
Aliaksandr Valialkin	493d8ec7e0	docs/vmalert.md: clarify docs regarding the support of recursive globs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4041	2023-05-09 22:49:37 -07:00
Aliaksandr Valialkin	e2358d3bd5	docs: clarify docs after `5ee344824f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183	2023-05-09 22:49:13 -07:00
Aliaksandr Valialkin	079875a127	app/vmagent/remotewrite: make more user-friendly the warning message about too small -remoteWrite.maxdiskUsagePerURL value This is a follow-up for `bc17f4828c` . While at it, document the change at docs/CHANGELOG.md . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195	2023-05-09 22:48:40 -07:00
Aliaksandr Valialkin	f35181a4f8	docs/managed-victoriametrics: consistently use `setup-notifications` prefix for images used at docs/managed-victoriametrics/setup-notifications.md This should simplify managment of the images belonging to the given docs. See docs/assets/README.md for details This is a follow-up for `4052c44ac1`	2023-05-09 22:47:59 -07:00
Aliaksandr Valialkin	22aeeeef3e	app/vmauth: merge `default_url` example into multi-url example in order to reduce the amounts of text to read for the user Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084 This is a follow-up for `041e188df8`	2023-05-09 22:47:11 -07:00
Aliaksandr Valialkin	8703b2fa87	app/vmselect: small cleanup after `4f3f9950d0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3807	2023-05-09 22:45:02 -07:00
Aliaksandr Valialkin	fbc28810b1	app/vmselect: small cleanup after `68e31a6000` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3811	2023-05-09 22:43:59 -07:00
Aliaksandr Valialkin	5dbaffe2c6	app/{vmselect,vmctl}: move ParseTime() to lib/promutils Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4091 This is a follow-up for `e2053baf32`	2023-05-09 22:42:35 -07:00
Aliaksandr Valialkin	3a23d90c3a	docs/CHANGELOG.md: typo fix after `45a551df9c`: 'this doc' -> 'this feature request'	2023-05-09 22:42:07 -07:00
Aliaksandr Valialkin	143c5bc936	app/vmauth/README.md: mention about ip filters and concurrency limiter at Security chapter	2023-05-09 22:40:46 -07:00
Aliaksandr Valialkin	fe58dd7e4c	app/vmauth: refer ip_filters option in example auth config Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491	2023-05-09 22:40:15 -07:00
Aliaksandr Valialkin	e55ddaa962	docs: document IP filters functionality in vmauth Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491 This is a follow-up for `2f08ed3be2`	2023-05-09 22:39:47 -07:00
Aliaksandr Valialkin	2dcd754dc2	app/vmauth: remove duplicate mentioning of -auth.config value in error message in logs on usuccessful load of -auth.config This is a follow-up for `25759082f4`	2023-05-09 22:38:05 -07:00
Aliaksandr Valialkin	8cca9ed0d9	all: update Go builder from Go1.20.3 to Go1.20.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved	2023-05-09 22:31:08 -07:00
Aliaksandr Valialkin	cb72449fc4	docs/CHANGELOG.md: document `c77385e78f`	2023-05-09 22:29:23 -07:00
Alexander Marshalov	de68e94c91	fixed `vm_promscrape_config_last_reload_successful` metric value recovery after successful reloading with unchanged content (#4260 ) (#4268 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-09 22:17:27 -07:00
Roman Khavronenko	54bfacdfde	vmalert: follow-up after `cae87da` (#4269 ) * vmalert: follow-up after `cae87da` `cae87da4bb` Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: update struct comments Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: rm typo Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-09 22:07:26 -07:00
Alexander Marshalov	65f4590403	update generated docs for operator (#4267 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-09 22:06:28 -07:00
Haleygo	d1c68888bd	vmalert: support reading rule from http url (#4212 ) vmalert: support reading rule's config from HTTP URL	2023-05-09 21:59:21 -07:00
Roman Khavronenko	4edb97f4da	app/vmalert: detect alerting rules which don't match any series at all (#4198 ) app/vmalert: detect alerting rules which don't match any series at all vmalert starts to understand /query responses which contain object: ``` "stats":{"seriesFetched": "42"} ``` If object is present, vmalert parses it and populates a new field `SeriesFetched`. This field is then used to populate the new metric `vmalert_alerting_rules_last_evaluation_series_fetched` and to display warnings in the vmalert's UI. If response doesn't contain the new object (Prometheus or VictoriaMetrics earlier than v1.90), then `SeriesFetched=nil`. In this case, UI will contain no additional warnings. And `vmalert_alerting_rules_last_evaluation_series_fetched` will be set to `-1`. Negative value of the metric will help to compile correct alerting rule in follow-up. Thanks for the initial implementation to @Haleygo See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4056 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-09 21:48:59 -07:00
Zakhar Bessarab	2856e15a6e	deployment/docker: allow overriding docker namespace (#4265 ) It makes it easier for users who build and self-host images to publish their images without changing tags manually. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-09 21:44:22 -07:00
Roman Khavronenko	3049754575	alerts: update TooHighMemoryUsage threshold (#4256 ) It appears that 90% usage for anonymous mem usage is already concerning. So we lowering the threshold to 80%. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-09 21:41:40 -07:00
Nikolay	8c9dc837b9	lib/storage: properly update link for entry at dateMetricID cache (#4258 ) previously during sync for mutable and immutable cache parts, link for hotEntry with current date may be not properly updated it corrupts cache for backfilling metrics and increased cpu load	2023-05-09 21:39:41 -07:00
Zakhar Bessarab	370a421ef4	lib/promscrape/discovery/kubernetes: follow-up for `d5e94721db` (#4255 ) - add changelog reference to an author - fix tests - add metadata to match Prometheus behavior Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-09 21:29:27 -07:00
Vasilchenko Anton	866dbee4e3	Add endpoint labels for pod targets discovered form endpoint but has different ports (#4253 ) Signed-off-by: Vasilchenko Anton <vasilchenko-as@yandex.ru>	2023-05-09 21:25:56 -07:00
Zakhar Bessarab	348693ff84	lib/storage: fix indexdb rotation infinite loop (#4249 ) When using `retentionTimezoneOffset` and having local timezone being more than 4 hours different from UTC indexdb retention calculation could return negative value. This caused indexdb rotation to get in loop. Fix calculation of offset to use `retentionTimezoneOffset` value properly and add test to cover all legit timezone configs. See: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-05-09 21:23:01 -07:00
Alexander Marshalov	26fc4afff8	added new consulagent service discovery (#3953 ) (#4217 )	2023-05-08 23:43:59 -07:00
Alexander Marshalov	f7dd084890	max value for `memory.allowedPercent` changed from 200 to 100 (#4171 ) (#4251 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 23:20:56 -07:00
Zakhar Bessarab	98e4265235	docs: changelog follow-up for `49b77ec01a` (#4250 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-08 23:20:09 -07:00
justcompile	44e929fbc6	squash commits (#4166 )	2023-05-08 23:18:08 -07:00
Nikolay	7bfa1d7d9e	lib/backup: fixes path generation for windows (#4133 ) replaces custom fsync function with standard Fsync methods for files. fixes pattern matching for parts and properly generate backup path for local fs. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-05-08 23:16:26 -07:00
Nikolay	5d0299ac19	lib/fs: do not panic at windows at dir deletion (#4132 ) Windows doesn't allow to remove dir with opened files. Usually it's a case for snapshots, hard cannot be removed if file is openned. With this change, dir will be renamed and properly deleted at the next process start. It's recommended to restart vmstorage/vmsingle for snapshots deletion completion periodically. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-05-08 23:11:55 -07:00
Roman Khavronenko	5bc8d8f290	vmselect: exit early from queue on context cancel (#4223 ) * vmselect: exit early from queue on context cancel When `-search.maxConcurrentRequests` is reached, vmselect puts request in the queue. It is expected, that requests in the queue will be processed as soon as it would be enough capacity to do so. However, it could happen that while request was waiting its turn, the client could have already cancel it (close the connection, or just close the tab with UI). In this case, we should de-queue such requests to avoid spending extra resources on them. Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmselect: address review comments Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 22:58:05 -07:00
Zakhar Bessarab	1b06af321f	lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints (#4235 ) * lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints Sets `__meta_kubernetes_endpoints_name` and `__meta_kubernetes_namespace` labels to all ports of pod. Prometheus sets those labels to all ports in pod (`0ab9553611/discovery/kubernetes/endpoints.go (L267C15-L269)`) even if port is not matching any service. See: #4154 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/kubernetes: fix test for updated discovery logic Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-08 22:15:37 -07:00
Artem Navoiev	f168da9524	remove information of releasing graphite render api from tip section as we released it in 1.90 Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 22:01:13 -07:00
Max Golionko	ef11e49d7b	add vmsingle filter for health alerts (#4238 )	2023-05-08 22:00:43 -07:00
Artem Navoiev	46886260be	prepare operator docs to migration Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 21:59:48 -07:00
Artem Navoiev	1e0dcfaae6	update managed docs - prepare for migration Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 21:58:53 -07:00
Roman Khavronenko	f247160699	docs: note automatic conversion to `ms` for influx protocol (#4224 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 21:55:58 -07:00
Artem Navoiev	1e15eea76a	add wight do trobubleshooting docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 21:54:48 -07:00
Artem Navoiev	283a75afba	prepare static docs to migration Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 21:54:20 -07:00
Aliaksandr Valialkin	7acc54025e	Revert "lib/streamaggr: discard samples with timestamps outside of aggregation interval (#4199 )" This reverts commit `9e99f2f5b3`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4068 Reason for revert: this breaks valid use cases: - If timestamps aren't specified in the incoming samples on purpose. For example, if stream aggregation is used as StatsD replacement. StatsD protocol has no timestamp concept for incoming samples. See https://github.com/b/statsd_spec - If all the samples must be aggregated, even if they contain stale timestamps. for example, if the stream aggregation produces some counter of some events, it may be better to count all the events even if they were delayed before being ingested into VictoriaMetrics. Is is also unclear how to determine whether the sample becomes stale. For example, if the aggregation interval equals to 1h, and the previous aggregation cycle just finished 10 minutes ago, what to do with the newly incoming sample with the timestamp 30 minutes older than the current time? The answer highly depends on the context, so it is unsafe to uncoditionally use a single logic for dropping the old samples here.	2023-05-08 21:50:19 -07:00
Roman Khavronenko	8f1372bd43	vmalert: fix API to return non-nil values (#4222 ) Properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. This improves compatibility with Grafana. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 21:47:51 -07:00
Aliaksandr Valialkin	1a7794735e	app/vmselect: fix the build after fb8889820aba710508033cbf6826eb63a357532a	2023-05-08 17:32:18 -07:00
Roman Khavronenko	c6511bc2d0	Revert "http server: limit max concurrent requests (#4185 )" (#4215 ) This reverts commit `77f76371` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 17:22:27 -07:00
Roman Khavronenko	1c3bf0d0d8	app/vmalert: follow-up after `6c322b4a00` (#4214 ) `6c322b4a00` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 17:20:49 -07:00
Haleygo	4b0db17bec	vmalert: allow configuring custom notifier headers per group (#4088 ) vmalert: allow configuring custom notifier headers per group	2023-05-08 17:07:44 -07:00
Zakhar Bessarab	52021713ec	lib/streamaggr: discard samples with timestamps outside of aggregation interval (#4199 ) * lib/streamaggr: discard samples with timestamps not matching aggregation interval Samples with timestamps lower than `now - aggregation_interval` are likely to be written via backfilling and should not be used for calculation of aggregation. See #4068 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/streamaggr: make log message more descriptive, fix imports Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-08 17:06:35 -07:00
Haleygo	4c3cb7a7ad	lib/opentsdbhttp: fix a typo preventing from using writeconcurrencylimiter (#4208 )	2023-05-08 16:33:03 -07:00
Zakhar Bessarab	19eaf17e11	app/vmalert: add support of recursive path globs for rules and templates (#4148 ) Supports using `` for `-rule` and `-rule.templates`: `dir//*.tpl` loads contents of dir and all subdirectories recursively. See: #4041 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-05-08 16:22:30 -07:00
Nikolay	cfa058dfec	lib/promscrape: adds filter for consul_sd_configs: (#4184 ) * lib/promscrape: adds filter for consul_sd_configs: it allows advanced filtering for consul service discovery requests https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183 * typo fix * removes deprecation mentions since it's not relevant * Update docs/CHANGELOG.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-05-08 16:14:15 -07:00
Max Golionko	0f14beff58	alerts: relax job filter to support job names created by VMOperator (#4203 )	2023-05-08 15:52:31 -07:00
Zakhar Bessarab	55d772ab39	app/vmalert: return an error when using `query` function in `-external.alert.source` flag (#4191 ) Templating of `-external.alert.source` is not expected to have access to the query which was causing runtime error when query function was passed as nil. See: #4181 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-05-08 15:48:16 -07:00
7840vz	d7157e582c	alerts: decrease severity to info for RecordingRulesNoData (#4089 )	2023-05-08 15:46:15 -07:00
Dmytro Kozlov	f425123116	app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB (#4196 ) * app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB * app/vmagent,lib/persistentqueue: linter fix * app/vmagent,lib/persistentqueue: fix comment	2023-05-08 15:45:21 -07:00
dmitryk-dk	f1717aa818	docs: fix type	2023-05-08 15:33:21 -07:00
dmitryk-dk	6168e7f95f	app/vmctl: fix comments	2023-05-08 15:32:24 -07:00
dmitryk-dk	aef925ea8a	app/vmctl: fix comments	2023-05-08 15:31:45 -07:00
dmitryk-dk	c2b58fd954	docs/managed-victoriametrics: change emails	2023-05-08 15:31:01 -07:00
dmitryk-dk	dbee230911	docs/managed-victoriametrics: add notifications setup	2023-05-08 15:30:21 -07:00
Alexander Marshalov	402d906d40	added `default_url` field in vmauth users config (#4084 ) (#4156 ) * added default url field in vmauth users config (#4084) --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 15:13:46 -07:00
Yury Molodov	ddc5197bce	vmui: add metric relabel debug (#3889 ) * feat: add metric relabel debug (#3807) * fix: add link to relabeling cookbook * lib/promrelabel: merge, fix conflicts * lib/promrelabel: fix diff * docs/vmui: add metric relabel playground --------- Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>	2023-05-08 14:59:35 -07:00
Alexander Marshalov	42b1e6443b	changelog for issue #4083 (#4197 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 14:41:39 -07:00
Yury Molodov	e0434f7b4e	vmui: display heatmap in the `Explore Metrics` (#4124 ) * feat: display heatmap in the explore metrics (#4111) * fix: correct calc step for heatmap * fix: remove spaces in the result of getDurationFromMilliseconds	2023-05-08 14:40:38 -07:00
Yury Molodov	3c45256736	vmui: add a comparison of data to the `Cardinality Explorer` (#4123 ) * feat: add button "show today" to date picker * feat: add comparison with the prev day (#3967) * vmui/docs: add comparison of data to cardinality page	2023-05-08 14:39:14 -07:00
Yury Molodov	382a2ff649	docs/vmui: fix CHANGELOG.md about WITH templates (#4194 )	2023-05-08 14:37:08 -07:00
Yury Molodov	de35cbf251	vmui: Integrate WITH template playground (#3831 ) * feat: add WithTemplate page * app/vmselect/prometheus: enable json mode for expand with expr API * app/vmselect/prometheus: enable CORS and add content type * feat: add api for expand with templates * fix: remove console from useExpandWithExprs * app/vmselect/prometheus: fix escaping * vmui: integrate WITH template * app/vmctl: check content type instead of form param * fix: add content-type for fetch with-exprs * fix: add a header to the server's response that allows the "Content-Type" header * app/vmctl: added comment and cleanup * app/vmctl: use format query param --------- Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>	2023-05-08 14:35:35 -07:00
Dmytro Kozlov	3da856af0c	app/vmctl: add support for the different time format in the native binary protocol (#4189 ) * app/vmctl: add support for the different time format in the native binary protocol * app/vmctl: update flag description, update CHANGELOG.md * app/vmctl: add comment to exported function	2023-05-08 14:21:59 -07:00
Alexander Marshalov	574a0559d5	added `unauthorized_user` field in vmauth users config (#4083 ) (#4157 ) added `unauthorized_user` field in vmauth users config (#4083) --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 13:36:54 -07:00
Roman Khavronenko	20b025dc88	http server: limit max concurrent requests (#4185 ) * lib/httpserver: introduce `-http.maxConcurrentRequests` command-line flag Introduce `-http.maxConcurrentRequests` command-line flag to protect VM components from resource exhaustion during unexpected spikes of HTTP requests. By default, the new flag's value is set to 0 which means no limits are applied. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/httpserver: mention http.maxConcurrentRequests in docs Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 13:13:58 -07:00
Alexander Marshalov	7faf45d7f9	doc improvements (#4172 ) (#4186 ) - added info about metric `vm_vminsert_metrics_read_total`, - small doc refactoring - and added make-command for running docs in docker. Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 13:11:57 -07:00
Yury Molodov	23ed0467a2	vmui: fix freeze when query regular with heatmap query (#4093 ) * fix: fix freeze when query regular with heatmap query * vmui/docs: fix freeze when query regular with heatmap query	2023-05-08 13:09:49 -07:00
Aliaksandr Valialkin	45ec3ac158	docs: document IP filters functionality in vmauth Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491 This is a follow-up for `2f08ed3be2`	2023-05-08 12:14:05 -07:00
Yury Molodov	46765c677d	vmui: fix bug where tenant list was not displayed (#4162 ) * fix: modify the condition for querying tenants * fix: change getTenantIdFromUrl output to string	2023-05-08 11:00:19 -07:00
Alexander Marshalov	f5981c1447	vmauth ip filters (refactoring) (#4059 ) Added ip filters (allow_list and deny_list) for enterprise-version of vmauth (#3491) --------- Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-05-08 10:57:33 -07:00
Artem Navoiev	624e13a1d6	move note about opensource of graphite in v1.90 release note Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 09:56:53 -07:00
Roman Khavronenko	e9ce67adb8	vmalert: retry datasource requests with EOF or unexpected EOF errors (#4146 ) * vmalert: retry datasource requests with EOF or unexpected EOF errors Retry failed read request on the closed connection one more time. This may improve rules execution reliability when connection between vmalert and datasource closes unexpectedly. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: fix old tests Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 09:49:49 -07:00
Zakhar Bessarab	79ee1749a1	lib/httpserver: add handler to serve `/robots.txt` and deny search indexing (#4143 ) This handler will instruct search engines that indexing is not allowed for the content exposed to the internet. This should help to address issues like #4128 when instances are exposed to the internet without authentication.	2023-05-08 09:46:34 -07:00
Balamurugan Krishnamoorthy (Bala)	2a45a85c35	Removed duplicate third-party article reference (#4142 ) "How do We Keep Metrics for a Long Time in VictoriaMetrics" article is referenced twice in "Third-party articles and slides about VictoriaMetrics" section	2023-05-08 09:44:19 -07:00
Artem Navoiev	dcac82638e	add graphite render api opensource to changelog Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-05-08 09:42:38 -07:00
Aliaksandr Valialkin	cd99f1fb3a	docs/CHANGELOG.md: document `c77385e78f`	2023-05-08 08:57:49 -07:00
Nikolay	82e2f19bc2	app/vminsert: correctly allocate buffer for storagenodes (#554 ) in case of dynamic discovery number of nodes may change and we have to allocate new buffer for this case otherwise vminsert may panic	2023-05-08 08:57:15 -07:00
Aliaksandr Valialkin	8b15f93426	lib/{mergeset,storage}: make mustReadPartNames() code more clear	2023-04-14 23:17:08 -07:00
Aliaksandr Valialkin	d739511f5b	lib/storage: replace OpenStorage() with MustOpenStorage() Callers of OpenStorage() log the returned error and exit. The error logging and exit can be performed inside MustOpenStorage() alongside with printing the stack trace for better debuggability. This simplifies the code at caller side.	2023-04-14 23:04:42 -07:00
Aliaksandr Valialkin	f26e480a77	lib/storage: fix a bug, which prevents from reading pre-v1.90.0 parts The bug has been introduced in `c0b852d50d`	2023-04-14 22:33:29 -07:00
Aliaksandr Valialkin	cf4701db65	lib/fs: add MustReadDir() function Use fs.MustReadDir() instead of os.ReadDir() across the code in order to reduce the code verbosity. The fs.MustReadDir() logs the error with the directory name and the call stack on error before exit. This information should be enough for debugging the cause of the error.	2023-04-14 22:11:40 -07:00
Aliaksandr Valialkin	0a11c46cd2	lib/storage: validate rows in partition.AddRows() only during tests	2023-04-14 20:53:05 -07:00
Aliaksandr Valialkin	292b6a851f	all: consistently use fs.MustClose() for closing lock files	2023-04-14 20:16:11 -07:00
Aliaksandr Valialkin	a7678350ad	lib/fs: convert CreateFlockFile to MustCreateFlockFile Callers of CreateFlockFile log the returned err and exit. It is better to log the error inside the MustCreateFlockFile together with the path to the specified directory and the call stack. This simplifies the code at the callers' side while leaving the debuggability at the same level.	2023-04-14 19:51:52 -07:00
Aliaksandr Valialkin	e2de5bf763	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart Callers of InitFromFilePart log the error and exit. It is better to log the error with the path to the part and the call stack directly inside the MustInitFromFilePart() function. This simplifies the code at callers' side while leaving the same level of debuggability.	2023-04-14 15:47:20 -07:00
Aliaksandr Valialkin	df99965564	lib/filestream: change Create() to MustCreate() Callers of this function log the returned error and exit. It is better logging the error together with the path to the filename and call stack directly inside the function. This simplifies the code at callers' side without reducing the level of debuggability	2023-04-14 15:14:24 -07:00
Aliaksandr Valialkin	0bbb281c3d	lib/filestream: transform Open() -> MustOpen() Callers of this function log the returned error and exit. Let's log the error with the path to the filename and call stack inside the function. This simplifies the code at callers' side without reducing the level of debuggability.	2023-04-14 15:04:54 -07:00
Aliaksandr Valialkin	ee8be138b9	lib/fs: improve error logging at ReaderAt.MustReadAt() - Add 'BUG:' prefix to error messages related to programming errors aka bugs. - Consistently log the path to the file in all the messages in order to improve debuggability.	2023-04-14 14:52:14 -07:00
Aliaksandr Valialkin	b80d93d4b2	lib/fs: substitute ReadFullData with MustReadData Callers of ReadFullData() log the error and then exit. So let's log the error with the path to the filename and the call stack inside MustReadData(). This simplifies the code at callers' side, while leaving the debuggability at the same level.	2023-04-14 14:40:58 -07:00
Aliaksandr Valialkin	36559dfec2	lib/fs: improve error logging inside MustWriteData Log the path to file on errors inside MustWriteData(). This improves debuggability of errors, which may occur inside MustWriteData().	2023-04-14 14:33:45 -07:00
Aliaksandr Valialkin	d544bfd73e	vendor: update github.com/VictoriaMetrics/metricsql from v0.56.1 to v0.56.2 This fixes panic when the duration in the query contains `M` suffix. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589	2023-04-14 14:06:47 -07:00
Aliaksandr Valialkin	67df75484f	lib/{mergeset,storage}: remove isInMerge flag from parts only when they werent removed yet from the list of active parts This prevents from possible panic during access to pw.p when it is set to nil at partWrapper.decRef() called inside swapSrcWithDstParts()	2023-04-14 00:16:18 -07:00
Aliaksandr Valialkin	2fd9fdacce	docs/vmctl.md: run `make docs-sync` after `2a5b9ff782`	2023-04-13 23:54:50 -07:00
Aliaksandr Valialkin	e6d10106f7	docs/CHANGELOG.md: move the bugfix description into the correct place This is a follow-up for `2a5b9ff782` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092	2023-04-13 23:52:01 -07:00
Dmytro Kozlov	10351a9dbb	app/vmctl: fix performance degradation, add flag to disable backoff policy (#4097 ) * app/vmctl: change api for getting metric names * app/vmctl: fix tests * app/vmctl: add flag to enable backoff policy, fix test, performance improvements * app/vmctl: use one http client * app/vmctl: made linter happy * app/vmctl: updated documentation and CHANGELOG.md * app/vmctl: cleanup * app/vmctl: rename flag * app/vmctl: cleanup * app/vmctl: fix comments * app/vmctl: fix metrics parser problem, improve tests	2023-04-13 23:51:11 -07:00
Aliaksandr Valialkin	7fb2b14ca0	docs/CHANGELOG.md: run at least 4 background mergers on systems with less than 4 CPU cores This reduces the probability of sudden spike in the number of small parts when all the background mergers are busy with big merges.	2023-04-13 23:37:05 -07:00
Aliaksandr Valialkin	8846ce5f1d	lib/{mergeset,storage}: make sure that getFlushToDiskDeadline() takes into account only in-memory parts	2023-04-13 23:17:24 -07:00
Aliaksandr Valialkin	f75b1b7a53	lib/fs: add Must prefix to CopyDirectory and CopyFile functions Callers of these functions log the returned error and then exit. Let's log the error with the call stack inside the function itself. This simplifies the code at callers' side, while leaving the same level of debuggability in case of errors.	2023-04-13 23:04:37 -07:00
Aliaksandr Valialkin	75b74aa837	lib/fs: rename SymlinkRelative to MustSymlinkRelative Callers of this function log the returned error and then exit. Let's log the error with the call stack inside the function itself. This simplifies the code at callers' side, while leaving the same level of debuggability in case of errors.	2023-04-13 22:53:11 -07:00
Aliaksandr Valialkin	624b86d065	lib/fs: rename HardLinkFiles to MustHardLinkFiles Callers of this function log the returned error and then exit. Let's log the error with the call stack inside the function itself. This simplifies the code at callers' side, while leaving the same level of debuggability in case of errors.	2023-04-13 22:49:38 -07:00
Aliaksandr Valialkin	c4638553a3	lib/fs: rename WriteFileAtomically to MustWriteAtomic Callers of this function log the returned error and exit. So let's just log the error with the given filepath and the call stack inside the function itself and then exit. This simplifies the code at callers' place while leaves the same level of debuggability in case of errors.	2023-04-13 22:43:30 -07:00
Aliaksandr Valialkin	aac3dccfd1	lib/fs: replace MkdirAllIfNotExist->MustMkdirIfNotExist and MkdirAllFailIfExist->MustMkdirFailIfExist Callers of these functions log the returned error and then exit. The returned error already contains the path to directory, which was failed to be created. So let's just log the error together with the call stack inside these functions. This leaves the debuggability of the returned error at the same level while allows simplifying the code at callers' side. While at it, properly use MustMkdirFailIfExist instead of MustMkdirIfNotExist inside inmemoryPart.MustStoreToDisk(). It is expected that the inmemoryPart.MustStoreToDick() must fail if there is already a directory under the given path.	2023-04-13 22:22:08 -07:00
Aliaksandr Valialkin	b4c330ea2b	lib/fs: rename MustWriteFileAndSync to MustWriteSync in order to improve readability a bit This is a follow-up for `2a8395be05`	2023-04-13 22:20:31 -07:00
Aliaksandr Valialkin	cdee2cfc5c	lib/{mergeset,storage}: remove unused `path` field from blockStreamWriter This is a follow-up after `42bba64aa7`	2023-04-13 22:20:02 -07:00
Aliaksandr Valialkin	1cda542c48	lib/fs: replace WriteFileAndSync with MustWriteAndSync When WriteFileAndSync fails, then the caller eventually logs the error message and exits. The error message returned by WriteFileAndSync already contains the path to the file, which couldn't be created. This information alongside the call stack is enough for debugging the issue. So just use log.Panicf("FATAL: ...") inside MustWriteAndSync(). This simplifies error handling at caller side a bit.	2023-04-13 22:17:34 -07:00
Aliaksandr Valialkin	eb7df27e20	lib/{mergeset,storage}: properly fsync part directory listing after writing in-memory part to disk This is a follow-up after `42bba64aa7` Previously the part directory listing was fsync'ed implicitly inside partHeader.WriteMetadata() by calling fs.WriteFileAtomically(). Now it must be fsync'ed explicitly. There is no need in fsync'ing the parent directory, since it is fsync'ed by the caller when updating parts.json file.	2023-04-13 21:21:46 -07:00
Aliaksandr Valialkin	13d2350e6a	lib/{mergeset,storage}: explicitly fsync the created part directory listing Previously the created part directory listing was fsynced implicitly when storing metadata.json file in it. Also remove superflouous fsync for part directory listing, which was called at blockStreamWriter.MustClose(). After that the metadata.json file is created, so an additional fsync for the directory contents is needed.	2023-04-13 21:07:33 -07:00
Aliaksandr Valialkin	cf53ce83a0	app/vmstorage: deprecate -bigMergeConcurrency command-line flag Improperly configured -bigMergeConcurrency command-line flag usually leads to uncontrolled growth of unmerged parts, which, in turn, increases CPU usage and query durations. So it is better deprecating this flag. In rare cases -smallMergeConcurrency command-line flag can be used instead for controlling the concurrency of background merges.	2023-04-13 20:42:22 -07:00
Aliaksandr Valialkin	e73dd1df2d	lib/{fs,persistentqueue}: use filepath.Join() instead of concatenating path parts with `/` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014	2023-04-13 20:14:07 -07:00
Aliaksandr Valialkin	e95e401e4d	app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71	2023-04-13 19:38:28 -07:00
Alexander Marshalov	e0a1b62569	Added extra information to docs about `total` output in stream aggregation. (#4130 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-04-13 19:37:05 -07:00
Zakhar Bessarab	217eea6e15	lib/backup/actions: store metadata(creation and completion time) in backup files (#4117 ) This makes it easier to understand exact point in time which is included in this backup. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-04-13 19:20:34 -07:00
Aliaksandr Valialkin	61839d4807	vendor: update github.com/valyala/gozstd from v1.19.0 to v1.19.1	2023-04-10 11:30:26 -07:00
Aliaksandr Valialkin	6e8c4ee95c	app/vmbackupmanager/README.md: sync with docs/vmbackupmanager.md after `4b2cc1b32c`	2023-04-10 10:52:34 -07:00
Aliaksandr Valialkin	b33dd2e577	docs/Single-server-VictoriaMetrics.md: document automatic switch from graph view to heatmap view for histogram buckets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384	2023-04-10 10:52:24 -07:00
Aliaksandr Valialkin	6c7087f65e	docs/Troubleshooting.md: mention that it is recommended to use default command-line flag values for VictoriaMetrics components	2023-04-10 10:52:10 -07:00
Aliaksandr Valialkin	4077b0fcb1	vendor: `make vendor-update`	2023-04-10 10:21:39 -07:00
Haleygo	7ee32ed06a	fix sort pendingDateMetricsIDs (#4102 )	2023-04-10 10:16:36 -07:00
Aliaksandr Valialkin	7f55afd699	deployment: update VictoriaMetrics from v1.89.1 to v1.90.0 See https://docs.victoriametrics.com/CHANGELOG.html#v1900	2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin	b21ac56894	docs/guides: update VictoriaMetrics from v1.89.1 to v1.90.0	2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin	2bbd8754bb	docs/CHANGELOG.md: formatting fix	2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin	64cf8fe492	app/vmctl/terminal: fix builds for GOOS=freebsd and GOOS=openbsd This is a follow-up for `8da9502df6`	2023-04-06 17:09:20 -07:00
Aliaksandr Valialkin	d15618f2ef	docs/CHANGELOG.md: cut v1.90.0 release	2023-04-06 16:16:52 -07:00
Aliaksandr Valialkin	26b361f4c3	app/vmselect/vmui: run `make vmui-update` after `01fc228fb0`	2023-04-06 15:11:54 -07:00
Dmytro Kozlov	8ec5e7f53a	app/vmctl: add multiple filters defined in `--vm-native-filter-match` flag to discovered metric names (#4063 ) * app/vmctl: add multiple filters defined in `--vm-native-filter-match` flag to discovered metric names * app/vmctl: fix comments * app/vmctl: move function buildMatchWithFilter to the correct place * app/vmctl: update CHANGELOG.md * app/vmctl: fix CI, remove error wrapping * app/vmctl: fix CI, simplify `Set()`	2023-04-06 15:11:53 -07:00
Yury Molodov	d9876b36b5	vmui: heatmap fixes (#4086 ) * fix: correct display of errors for query * fix: change the logic of histogram detection * feat: hide empty buckets from the graph * fix: revert server url	2023-04-06 15:11:26 -07:00
Aliaksandr Valialkin	ff1891397a	docs/CHANGELOG.md: document the bugfix, which remove unneeded logger.Errorf() call during stream aggregation with the enabled deduplication This is a follow-up for `ff72ca14b9`	2023-04-06 15:01:25 -07:00
Aliaksandr Valialkin	a241485262	app/vmselect/vmui: run `make vmui-update` after `a1601929ec`	2023-04-06 03:20:16 -07:00
Aliaksandr Valialkin	eca03d1f55	docs/CHANGELOG.md: document that VictoriaMetrics for Windows cannot delete snapshots See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183	2023-04-06 03:16:20 -07:00
Aliaksandr Valialkin	c43dc072c4	docs/CHANGELOG.md: document v1.79.12	2023-04-06 03:10:12 -07:00
Aliaksandr Valialkin	c92731de93	docs/CHANGELOG.md: document v1.87.5	2023-04-06 00:44:30 -07:00
Timur Bakeyev	f21ea62ae1	Fix cut-n-paste error (#4079 ) It seems that VMServiceScrape description was c-n-p from vmselect one into all other resources.	2023-04-06 00:44:29 -07:00
Yury Molodov	2263c412da	fix: correct the description of shortcut keys (#4057 )	2023-04-05 22:20:12 -07:00
Zakhar Bessarab	0384b83736	docs: fix example operator spec for vmbackupmanager restore usage (#4074 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-04-05 22:17:05 -07:00
Yury Molodov	7871ee0e43	vmui: implement heatmap improvements (#4078 ) * fix: disabled limits for histogram * fix: add sorted buckets by upper bound * refactor: move line chart components to folder * feat: implement heatmap improvements (https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384#issuecomment-1484023162) * app/vmselect/vmui: `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-04-05 22:15:23 -07:00
Aliaksandr Valialkin	bf545fcc14	lib/encoding: fix test after `4725549cb2`	2023-04-05 21:38:48 -07:00
Aliaksandr Valialkin	180a5be623	vendor: update github.com/klauspost/compress from v1.16.3 to v1.16.4 See https://github.com/klauspost/compress/releases/tag/v1.16.4	2023-04-05 21:27:13 -07:00
Aliaksandr Valialkin	93cfb448c7	vendor: update github.com/valyala/gozstd from v1.18.0 to v1.19.0	2023-04-05 20:55:57 -07:00
Timur Bakeyev	210216a1e2	Fix reference to the imagepullsecrets description (#4080 ) Looks like the original document has moved to the https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod. Alternatively, it could be that https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ describes the meaning of the parameter in more detail.	2023-04-05 19:57:22 -07:00
Aliaksandr Valialkin	ac2fbf0a9c	docs/Troubleshooting.md: add missing `help` word after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg	2023-04-05 14:28:19 -07:00
Aliaksandr Valialkin	e74c54aa86	docs/Troubleshooting.md: another typo fixes after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg	2023-04-05 14:15:00 -07:00
Aliaksandr Valialkin	cd2ed803c2	docs/Troubleshooting.md: fix formatting after `c7bcf750c2`	2023-04-05 13:46:26 -07:00
Aliaksandr Valialkin	775eb1245d	docs/Troubleshooting.md: fix a typo in the link after `c7bcf750c2`	2023-04-05 13:40:32 -07:00
Aliaksandr Valialkin	52c46f49e1	all: update Go builder from Go1.20.2 to Go1.20.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved	2023-04-05 13:38:44 -07:00
Artem Navoiev	c5397c0b77	update changelog Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-04-05 13:31:48 -07:00
Zakhar Bessarab	303d49a4c2	deployment/docker: update Grafana URLs to match latest format (#4060 ) See: #4019 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-04-05 13:31:29 -07:00
Zakhar Bessarab	54edd6992a	app/vmalert: update Grafana URLs to match latest format (#4061 ) See: #4019 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-04-05 13:31:06 -07:00
Artem Navoiev	1b05896ac5	fix closing divs in docs Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-04-05 13:30:14 -07:00
Aliaksandr Valialkin	ca4f94c704	docs/Troubleshooting.md: add General troubleshooting checklist This checklist helps searching for the infromation related to some issue / question about VictoriaMetrics	2023-04-05 13:29:12 -07:00
Aliaksandr Valialkin	52734c71fc	lib/storage: use shorter code after `03bde173b7`	2023-04-02 21:35:34 -07:00
faceair	03bde173b7	lib/storage: fix reuse pendingMetricRow (#4049 )	2023-04-02 21:28:43 -07:00
Aliaksandr Valialkin	286697b8b6	docs/CHANGELOG.md: document `edb45d7fc1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4013	2023-04-02 21:26:28 -07:00
faceair	a4b4bda166	lib/storage: remove unused code (#4050 )	2023-04-02 21:23:24 -07:00
Aliaksandr Valialkin	fa2ba7b07b	app/vmselect/vmui: run `make vmui-update` after `edb45d7fc1`	2023-04-02 21:22:17 -07:00
Yury Molodov	8bc5e83872	feat: add accept/cancel buttons for settings (#4013 ) (#4052 )	2023-04-02 21:22:11 -07:00
Aliaksandr Valialkin	02ceebccc0	lib/promscrape: do not re-use previously loaded scrape targets on failed attempt to load updated scrape targets at file_sd_configs The logic employed for re-using the previously loaded scrape target was broken initially. The commit `cc0427897c` tried to fix it, but the new logic became too complex and fragile. So it is better to just remove this logic, since the targets from temporarily broken file should be eventually loaded on next attempts every -promscrape.fileSDCheckInterval This also allows removing fragile hacks around __vm_filepath label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989	2023-04-02 21:11:12 -07:00
Dmytro Kozlov	6f0512a81c	lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read (#4027 ) * lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read * lib/promscrape: clarified comment * lib/promscrape: made better approach to handle a problem with growing []ScrapeWork on each error when loading config lib/promscrape: added CHANGELOG.md * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-04-02 21:11:10 -07:00
Aliaksandr Valialkin	7b10af4846	app/vmselect/vmui: run `make vmui-update` after `42087518ba`	2023-04-01 00:41:03 -07:00
Yury Molodov	17b4fc9470	vmui: tips for working with the graph and legend (#4045 ) * feat: add tips for working with the graph and legend * feat: add the ability to collapse the legend * vmui/docs: add the ability to collapse the legend --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-04-01 00:41:01 -07:00
Aliaksandr Valialkin	0275acc43c	vendor: `make vendor-update`	2023-04-01 00:00:00 -07:00
Yury Molodov	9ff8acec60	vmui: add a tip for JSON and Table tabs (#4000 ) * feat: add a tip for JSON and Table tabs * feat: add Hyperlink component * fix: update Hyperlink * fix: update link to instant query	2023-03-31 23:59:59 -07:00
Roman Khavronenko	5f95f9d453	lib/storage: check for free disk space before opening tables (#4035 ) * lib/storage: check for free disk space before opening tables We check for free disk space before call to `openTable`, so `Storage` can be set to ReadOnly before mergeWorkers start. Before the change, there was a chance that merges will start even if Storage has to start in ReadOnly mode because of `-storage.minFreeDiskSpaceBytes` limit. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: chore Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/storage.go --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-31 23:50:56 -07:00
Aliaksandr Valialkin	db8fda4ec6	app/vmselect/graphite: open source Graphite Render API	2023-03-31 23:37:40 -07:00
Aliaksandr Valialkin	43b431d322	deployment/docker: update base Docker image from Alpine 3.17.2 to Alpine 3.17.3 This fixes security issues from https://alpinelinux.org/posts/Alpine-3.17.3-released.html This is a follow-up for `59c350d0d2`	2023-03-31 22:54:48 -07:00
Aliaksandr Valialkin	29f376e916	lib/fs: follow-up for `ec45f1bc5f` Properly close response body before checking for the response code. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034	2023-03-31 22:54:33 -07:00
Aliaksandr Valialkin	dad13c0a91	lib/streamaggr: follow-up for `ff72ca14b9` - Make sure that the last successfully loaded config is used on hot-reload failure - Properly cleanup resources occupied by already initialized aggregators when the current aggregator fails to be initialized - Expose distinct vmagent_streamaggr_config_reload* metrics per each -remoteWrite.streamAggr.config This should simplify monitoring and debugging failed reloads - Remove race condition at app/vminsert/common.MustStopStreamAggr when calling sa.MustStop() while sa could be in use at realoadSaConfig() - Remove lib/streamaggr.aggregator.hasState global variable, since it may negatively impact scalability on system with big number of CPU cores at hasState.Store(true) call inside aggregator.Push(). - Remove fine-grained aggregator reload - reload all the aggregators on config change instead. This simplifies the code a bit. The fine-grained aggregator reload may be returned back if there will be demand from real users for it. - Check -relabelConfig and -streamAggr.config files when single-node VictoriaMetrics runs with -dryRun flag - Return back accidentally removed changelog for v1.87.4 at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3639	2023-03-31 22:54:10 -07:00
Max Golionko	a62f3034db	fix: app/vmui/Dockerfile-web to reduce vulnerabilities (#4044 ) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-ALPINE317-OPENSSL-3368755 - https://snyk.io/vuln/SNYK-ALPINE317-OPENSSL-3368755 - https://snyk.io/vuln/SNYK-ALPINE317-OPENSSL-5291795 - https://snyk.io/vuln/SNYK-ALPINE317-OPENSSL-5291795 Co-authored-by: snyk-bot <snyk-bot@snyk.io>	2023-03-31 22:44:33 -07:00
Zakhar Bessarab	743d1e6536	docs/vmctl: add examples of URLs used for migration in different modes (#4042 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-03-31 22:44:12 -07:00
Roman Khavronenko	0bde9722ed	vmalert: use `missingkey=zero` for templating (#4040 ) Replace empty labels with "" instead of "<no value>" during templating, as Prometheus does. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-31 22:43:39 -07:00
Zakhar Bessarab	46c8be4f98	lib/fs: verify response code when reading configuration over HTTP (#4036 ) Verifying status code helps to avoid misleading errors caused by attempt to parse unsuccessful response. Related issue: #4034 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-03-31 22:33:53 -07:00
Daria Karavaieva	83213e6786	Vmanomaly guide index fix (#4029 ) * name and scrutture change * fix indexing * index fix * name change * line separator fix	2023-03-31 22:32:15 -07:00
Alexander Marshalov	8c14d17694	added hot reload support for stream aggregation configs (#3969 ) (#3970 ) added hot reload support for stream aggregation configs (#3969) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-31 22:31:38 -07:00
Eliran Barnoy	7c70fb0fb9	Fix operator links to include VMPrometheusConverter for added visibility	2023-03-30 12:33:04 -07:00
Aliaksandr Valialkin	85ca077a88	lib/flagutil: ArrayString: support commas inside quoted strings and inside `[]`, `{}` and `()` braces Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915	2023-03-28 21:25:07 -07:00
Aliaksandr Valialkin	957d64e302	docs: mention that VictoriaMetrics rounds time range to UTC days at /api/v1/labels, /api/v1/label/.../values and /api/v1/series handlers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3107	2023-03-28 17:01:09 -07:00
Aliaksandr Valialkin	fd7efad69f	lib/persistentqueue: typo fix after `aea6df8197`	2023-03-27 20:05:51 -07:00
Aliaksandr Valialkin	6f5bbf096a	app/vmagent/remotewrite: cosmetic updates after `f3a51e8b1d` - Compare directory names instead of paths to directory when determining which persistent queues must be deleted This is less error-prone solution, since paths to the same directory can differ, which could lead to accidental directory removal for the existing -remoteWrite.url - Log the `removed %d dangling queues` message when at least a single queue has been removed - Consistently use filepath.Join() for creating paths to persistent queues. This is needed for Windows support (see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 ) - Clarify the description of the change at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014	2023-03-27 18:38:53 -07:00
Zakhar Bessarab	6ed6eb0c4c	app/vmagent: add `-remoteWrite.removeDanglingQueues` flag (#4017 ) * app/vmagent: add `-remoteWrite.removeDanglingQueues` flag which allows to automatically remove dangling persistent queue contents Related issue: #4014 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmagent: address review feedback - remove persistent queues files by default - rename `remoteWrite.removeDanglingQueues` to `remoteWrite.keepDanglingQueues` - update docs to reflect changed behaviour Related issue: #4014 * Apply suggestions from code review --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-27 18:38:51 -07:00
Nikolay	b38a145cfd	app/vmselect: properly remove temp files at windows system (#4020 ) With non-posix compliant systems it's not possible to remove unclosed files. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-27 18:10:44 -07:00
Aliaksandr Valialkin	54b9537a76	app/vmselect/promql: follow-up for `79e1c6a6fc` - Document the fix at docs/CHANGELOG.md - Add tests with multiple adjancent zero buckets - Simplify the fix a bit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021	2023-03-27 18:04:30 -07:00
Ze'ev Klapow	680a661ec0	fix le buckets when adjacent vmrange is empty (#4021 ) There is a bug here where if you have a single bucket like: foo{vmrange="4.084e+02...4.642e+02"} 2 123 The expected output is three le encoded buckets like: foo{le="4.084e+02"} 0 123 foo{le="4.642e+02"} 2 123 foo{le="+Inf"} 2 123 This correctly encodes the start and end of the vmrange. If however, the input contains the previous bucket, and that bucket is empty then you only get the end le and +Inf out currently, i.e: foo{vmrange="7.743e+05...8.799e+05"} 5 123 foo{vmrange="6.813e+05...7.743e+05"} 0 123 results in: foo{le="8.799e+05"} 5 123 foo{le="+Inf"} 5 123 This causes issues when you go to compute a quantile because this means that the assumed lower bound of the buckets is 0 and this we interpolate between 0->end rather than the vmrange start->end as expected.	2023-03-27 18:04:29 -07:00
Aliaksandr Valialkin	a3ae3268a1	vendor: `make vendor-update`	2023-03-27 15:28:11 -07:00
Aliaksandr Valialkin	9387793f47	app/vmselect: follow-up for `10ab086366` - Expose stats.seriesFetched at `/api/v1/query_range` responses too for the sake of consistency. - Initialize QueryStats when it is needed and pass it to EvalConfig then. This guarantees that the QueryStats is properly collected when the query contains some subqueries.	2023-03-27 15:11:42 -07:00
Roman Khavronenko	10ab086366	app/vmselect: export `seriesFetched` stat for /query responses (#3925 ) The change adds a new field `seriesFetched` to EvalConfig object. Since EvalConfig object can be copied inside `Exec`, `seriesFetched` is a pointer which can be updated by all copied objects. The reason for having stats is that other components, like vmalert, could benefit from this information. Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-27 08:51:33 -07:00
Yury Molodov	86a98fa131	vmui: heatmap (#3780 ) * fix: add stroke and font for all axes * feat: add util for generate gradient * feat: add heatmap plugin * feat: add heatmap legend * feat: add heatmap graph (#3384) * vmui: add heatmap graph (#3384) * feat: add convert Prometheus to VictoriaMetrics histogram * fix: prevent re-render graph * feat: reset step for heatmap * feat: normalize heatmap data * fix: format heatmap legend * wip * app/vmselect/vmui: run `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-26 00:31:21 -07:00
Aliaksandr Valialkin	229b39ac7d	docs/MetricsQL.md: quote min, max and avg args for rollup_*() functions in order to reduce the level of confusion when users try to pass the second argument to these functions	2023-03-26 00:02:01 -07:00
Aliaksandr Valialkin	84b3b0fac2	docs/CHANGELOG.md: document v1.87.4 LTS release	2023-03-25 22:44:21 -07:00
Aliaksandr Valialkin	db3bcbe56a	app/vmselect/netstorage: reduce the contention at fs.ReaderAt stats collection on systems with big number of CPU cores This optimization is based on the profile provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966#issuecomment-1483208419	2023-03-25 16:38:39 -07:00
Aliaksandr Valialkin	a2ecf4fa4a	app/vmselect/netstorage: document why runtime.Gosched() is removed at `28f054bb00` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966	2023-03-25 16:38:28 -07:00
Zakhar Bessarab	16f3b279a2	vmselect/netstorage: remove direct calls to `Gosched` to reduce amount of locks for global scope using `runtime.Gosched` requires acquiring global lock to check if there are any other goroutines to perform tasks. with the latest versions of runtime it can pause running goroutines automatically without requiring to call `Gosched` directly. Updates #3966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-03-25 16:37:58 -07:00
Aliaksandr Valialkin	ef4e5e4fc7	Makefile: add missing vmcluster-windows-amd64 target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 15:34:21 -07:00
Aliaksandr Valialkin	3698994953	app/{vmbackup,vmrestore}: publish vmbackup and vmrestore binaries for Windows Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 15:09:41 -07:00
Aliaksandr Valialkin	f6c36d5dfd	lib/storage: consistently use OS-independent separator in file paths This is needed for Windows support, which uses `\` instead of `/` as file separator Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 14:34:36 -07:00
Aliaksandr Valialkin	97b1e11612	lib/mergeset: consistently use OS-independent separator in file paths This is needed for Windows support, which uses `\` instead of `/` as file separator Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 14:34:33 -07:00
Aliaksandr Valialkin	1d9a461c23	all: follow-up after `34634ec357` - Use windows.FlushFileBuffers() instead of windows.Fsync() at streamTracker.adviseDontNeed() for consistency with implementations for other architectures. - Use filepath.Base() instead of filepath.Split(), since the dir part isn't used. This simplifies the code a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 12:00:48 -07:00
Nikolay	d231cefe25	lib/fs: adds memory map for windows (#3988 ) This is a follow-up for `43b24164ef` * lib/fs: adds memory map for windows it should improve performance for file reading * lib/storage: replace '/' with os specific separator it must fix an errors for windows * lib/fs: mention windows fsync support * lib/filestream: adds fdatasync for windows writes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-25 12:00:44 -07:00
Aliaksandr Valialkin	740fa57fdc	app/vmselect/promql: typo fix after `e7f46a0aab`	2023-03-24 23:47:11 -07:00
Aliaksandr Valialkin	7aff6f872f	app/vmselect/promql: follow-up for `7205c79c5a` - Allocate and initialize seriesByWorkerID slice in a single go instead of initializing every item in the list separately. This should reduce CPU usage a bit. - Properly set anti-false sharing padding at timeseriesWithPadding structure - Document the change at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966	2023-03-24 23:39:43 -07:00
Zakhar Bessarab	fec87e3ada	app/vmselect/promql: use lock-less approach to gather results of parallel processing for `evalRollup` funcs (#4004 ) vmselect/promql: refactor `evalRollupNoIncrementalAggregate` to use lock-less approach for parallel workers computation Locking there is causing issues when running on highly multi-core system as it introduces lock contention during results merge. New implementation uses lock less approach to store results per workerID and merges final result in the end, this is expected to significantly reduce lock contention and CPU usage for systems with high number of cores. Related: #3966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * vmselect/promql: add pooling for `timeseriesWithPadding` to reduce allocations Related: #3966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * vmselect/promql: refactor `evalRollupFuncWithSubquery` to avoid using locks Uses same approach as `evalRollupNoIncrementalAggregate` to remove locking between workers and reduce lock contention. Related: #3966 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-03-24 23:39:41 -07:00
Aliaksandr Valialkin	5c39b19acd	app/vmbackup: simplify code a bit after `5ba347bd2c` Unconditionally call deleteSnapshot() func just after making the snapshot, either successful or unsuccessful Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2055	2023-03-24 22:09:19 -07:00
Zakhar Bessarab	3f38ed3171	app/vmbackup: delete created snapshot in case of error during backup (#4008 ) Related issue: #2055 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-24 22:09:17 -07:00
Aliaksandr Valialkin	196f7398e0	vendor: run `make vendor-update`	2023-03-24 18:08:52 -07:00
Aliaksandr Valialkin	b9632023c4	app/vmselect/vmui: run `make vmui-update` after `dc2c712a29`	2023-03-24 18:08:51 -07:00
Aliaksandr Valialkin	181e877092	docs/CHANGELOG.md: cosmetic fixes: remove trailing whitespace and consistently use `-flag` instead of `--flag`	2023-03-24 17:57:23 -07:00
Alexander Marshalov	0301b5018e	allowed using dashes and dots in environment variables names (#4009 ) * allowed using dashes and dots in environment variables names for templating config files with envtemplate (#3999) Signed-off-by: Alexander Marshalov <_@marshalov.org> * Apply suggestions from code review --------- Signed-off-by: Alexander Marshalov <_@marshalov.org> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-24 17:57:19 -07:00
Aliaksandr Valialkin	c54b8acba2	docs/vmauth.md: follow-up for `36edba9bfb` - Document `-configCheckInterval` command-line flag in `quick start` section - Clarify the addition of `-configCheckInterval` at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3990	2023-03-24 17:56:59 -07:00
Aliaksandr Valialkin	9d19c2d89b	docs/vmagent.md: clarify that there is no need to specify multiple -remoteWrite.url options when writing data to a single VictoriaMetrics cluster when data replication is needed Also add a link to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format from `getting started` section, so users could quickly find how to write data to VictoriaMetrics cluster	2023-03-24 17:56:31 -07:00
Roman Khavronenko	a09dabc78f	vmalert: add anchor char to Group's link (#4006 ) This should help users to see that Group's name is clickable and used for anchoring. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-24 17:56:04 -07:00
Roman Khavronenko	ec6a20880c	vmalert: mention VMUI example for alert's source (#4005 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-24 17:55:30 -07:00
Roman Khavronenko	b866f334ed	docs: mention cluster URL for exporting series (#4002 ) docs: mention cluster URL for exporting series Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-24 17:54:57 -07:00
Daria Karavaieva	53c3d6137f	Vmanomaly guide (#3834 ) Setting up VMAnomaly on NodeExporter metrics with VictoriaMetrics and AlertManager. * vmanomaly-guide-draft * aletr graphs and description * readme vmanomaly tutorial * Added back fit_every param for performance * vmanomaly guide fixes * added spaces div * spaces + resize image * alert example grammar * quotation marks * docker link * typo fixed * more links * reader section rephrased * label change * lower case for grafana service * lower case for vm service * yaml markdown --------- Co-authored-by: Dima Lazerka <dima@victoriametrics.com>	2023-03-24 17:54:10 -07:00
Dmytro Kozlov	fe7acd5e8a	docs: follow up after `dc2c712a29` (#4001 )	2023-03-24 16:41:42 -07:00
Dmytro Kozlov	352dbd7e08	app/vmui: update cardinality page (#3986 ) vmui: update cardinality page --------- Co-authored-by: Yury Moladau <yurymolodov@gmail.com>	2023-03-24 13:34:01 -07:00
Yury Molodov	5f77efa915	vmui: display errors for each query individually (#3987 ) (#3994 )	2023-03-24 13:26:43 -07:00
Alexander Marshalov	b5027cff9c	added configCheckInterval flag for vmauth (#3990 ) (#3991 ) * added configCheckInterval flag for vmauth (#3990) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-24 13:25:07 -07:00
Nikolay	9bb83cafa4	lib/netutil: log only parsing errors for proxy-protocol (#3985 ) * lib/netutil: log only parsing errors for proxy-protocol Previosly every error was logged. With configured TCP health checks at load-balancer or kubernetes, vmauth spams a lot of false positive error message into logs * Update docs/CHANGELOG.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update lib/netutil/tcplistener.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-03-21 10:23:08 -07:00
Dmytro Kozlov	4ba237ec14	app/vmctl: follow up after `aed59b9029` (#3983 )	2023-03-21 09:26:26 -07:00
Aliaksandr Valialkin	8ed9295109	docs/vmagent.md: mention in docs that the target relabel debug page shows target url now	2023-03-20 22:20:13 -07:00
Dmytro Kozlov	85b01c4aa7	lib/promrelabel: make target url from labels on target relabel page (#3882 ) * lib/promrelabel: make target url from labels on target relabel page * wip --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-20 22:08:39 -07:00
Aliaksandr Valialkin	79d8f0e7c6	app/vmselect/promql: pass workerID to the callback inside doParallel() This opens the possibility to remove tssLock from evalRollupFuncWithSubquery() in the follow-up commit from @zekker6 in order to speed up the code for systems with many CPU cores. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966	2023-03-20 20:57:34 -07:00
Aliaksandr Valialkin	e749a015a9	app/vmselect/promql: fix TestIncrementalAggr test on systems less than 3 CPU cores This is a follow-up for `4856a4cf5a`	2023-03-20 20:37:44 -07:00
Aliaksandr Valialkin	08da383eac	app/vmselect/netstorage: reduce the number of calls to runtime.Gosched() at timeseriesWorker() and unpackWorker() Call runtime.Gosched() only when there is a work to steal from other workers. Simplify the timeseriesWorker() and unpackWroker() code a bit by inlining stealTimeseriesWork() and stealUnpackWork(). This should reduce CPU usage when processing queries on systems with big number of CPU cores. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966	2023-03-20 20:32:56 -07:00
Aliaksandr Valialkin	18af01c387	app/vmselect: optimize incremental aggregates a bit Substitute sync.Map with an ordinary slice indexed by workerID. This should reduce the overhead when updating the incremental aggregate state	2023-03-20 15:42:13 -07:00
Aliaksandr Valialkin	7a1e2f49cc	app/vmselect/vmui: `make vmui-update` after `d4525bd2d0`	2023-03-20 14:35:17 -07:00
Aliaksandr Valialkin	1580d8bda7	docs/CHANGELOG.md: cosmetic fixes	2023-03-20 14:33:46 -07:00
Aliaksandr Valialkin	531b35b6c0	docs/Troubleshooting.md: document an additional case, which could result in slow inserts If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series, then vm_slow_row_inserts_total` metric is increased. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183	2023-03-20 14:33:27 -07:00
Roman Khavronenko	6a7de761f4	vmalert: support logs suppressing during config reloads (#3973 ) * vmalert: support logs suppressing during config reloads The change is mostly required for ENT version of vmalert, since it supports object-storage for config files. Reading data from object storage could be time-consuming, so vmalert emits logs to track the progress. However, these logs are mostly needed on start or on manual config reload. Printing these logs each time `rule.configCheckInterval` is triggered would too verbose. So the change allows to control logs emitting during config reloads. Now, logs are emitted during start up or when SIGHUP is receieved. For periodicall config checks logs emitted by config pkg are suppressed. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: review fixes Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-20 14:25:26 -07:00
Dmytro Kozlov	aed59b9029	app/vmctl: automatically check tty (#3938 ) app/vmctl: automatically detect if TTY is available	2023-03-20 14:14:43 -07:00
Yury Molodov	95b60c2777	vmui: support for drag'n'drop in the "Trace analyzer" page (#3971 ) vmui: add drag-and-drop support for the trace analyzer page	2023-03-20 14:09:45 -07:00
dependabot[bot]	ba167df617	build(deps): bump actions/setup-go from 3 to 4 (#3962 ) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-03-20 14:08:28 -07:00
Yury Molodov	b66953d8e1	vmui: improve usability of date/time picker (#3968 ) * vmui: allow manually set input date and time * vmui/docs: improve usability of date/time picker	2023-03-20 13:57:47 -07:00
Dmytro Kozlov	693a3de0a6	lib/storage: fix collect downsampling metrics (#489 ) * lib/storage: fix downsampling * lib/storage: update logic * lib/storage: fix comments, removed unneeded check	2023-03-19 23:30:00 -07:00
Aliaksandr Valialkin	fc3d826d7f	all: add Windows build for VictoriaMetrics This commit changes background merge algorithm, so it becomes compatible with Windows file semantics. The previous algorithm for background merge: 1. Merge source parts into a destination part inside tmp directory. 2. Create a file in txn directory with instructions on how to atomically swap source parts with the destination part. 3. Perform instructions from the file. 4. Delete the file with instructions. This algorithm guarantees that either source parts or destination part is visible in the partition after unclean shutdown at any step above, since the remaining files with instructions is replayed on the next restart, after that the remaining contents of the tmp directory is deleted. Unfortunately this algorithm doesn't work under Windows because it disallows removing and moving files, which are in use. So the new algorithm for background merge has been implemented: 1. Merge source parts into a destination part inside the partition directory itself. E.g. now the partition directory may contain both complete and incomplete parts. 2. Atomically update the parts.json file with the new list of parts after the merge, e.g. remove the source parts from the list and add the destination part to the list before storing it to parts.json file. 3. Remove the source parts from disk when they are no longer used. This algorithm guarantees that either source parts or destination part is visible in the partition after unclean shutdown at any step above, since incomplete partitions from step 1 or old source parts from step 3 are removed on the next startup by inspecting parts.json file. This algorithm should work under Windows, since it doesn't remove or move files in use. This algorithm has also the following benefits: - It should work better for NFS. - It fits object storage semantics. The new algorithm changes data storage format, so it is impossible to downgrade to the previous versions of VictoriaMetrics after upgrading to this algorithm. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin	d2f85816ea	lib/{mergeset,storage}: prevent from long wait time when creating a snapshot under high data ingestion rate Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3873	2023-03-19 00:19:02 -07:00
Aliaksandr Valialkin	0627b845be	deployment/docker/Makefile: properly add amd64 suffix to windows binary names	2023-03-19 00:18:58 -07:00
Aliaksandr Valialkin	3473e83246	deployment/docker/Makefile: build CGO-enabled vmagent for GOARCH=arm64 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271 This is a follow-up for 565497fb074321caedea38d5151044d98d92d759	2023-03-18 23:15:38 -07:00
Aliaksandr Valialkin	f54ac85133	SECURITY.md: update the list of VictoriaMetrics versions, which support security updates	2023-03-18 12:28:26 -07:00
Aliaksandr Valialkin	8aeee8bcca	lib/{fs,mergeset,storage}: substitute os.Open()+os.File.Readdir() with os.ReadDir() This simplifies code a bit	2023-03-17 21:03:52 -07:00
Aliaksandr Valialkin	aeeab74388	app/vmctl: drop integration tests from cluster branch, since they expect single-node VictoriaMetrics This is a follow-up for `235477628e`	2023-03-17 16:43:58 -07:00
Alexander Marshalov	31005f4980	updated api doc for operator (#3972 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-17 16:17:17 -07:00
Roman Khavronenko	0ac57ef5b9	Vmalert tests (#3975 ) * vmalert: add tests for notifier pkg * vmalert: add tests for remotewrite pkg * vmalert: add tests for template functions * vmalert: add tests for web pages * vmalert: fix int overflow in tests Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-17 16:16:13 -07:00
Zakhar Bessarab	c397e56769	doc/vmgateway-grafana-openid-guide: fix formatting, add reproducible example and example results (#3964 )	2023-03-17 16:15:45 -07:00
Roman Khavronenko	66c5ddf2ad	alerts: add `TooManyTSIDMisses` alerting rule (#3959 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502#issuecomment-1358374954 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-17 16:14:50 -07:00
Aliaksandr Valialkin	c9e30dd892	all: typo fix in the same way as in `e566d49e3a`: 8248 -> 8428	2023-03-16 22:08:26 -07:00
oliverpool	8c708ca1e9	app/vmselect/promql: add test to ensure 8-byte alignment (#3948 ) See `0af9e2b693`	2023-03-16 22:07:13 -07:00
Artem Navoiev	7e6be18c3d	managed quickstart fix typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-16 22:04:03 -07:00
Artem Navoiev	208aa85362	fix anchor after chaning manager quick start Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-16 22:03:46 -07:00
Aliaksandr Valialkin	46521e20c8	vendor: `make vendor-update`	2023-03-15 13:24:22 -07:00
Nikolay	a61106d965	Vmagent kafka updates (#535 ) * app/vmagent: allow vm proto for kafka consumer and producer it should reduce network usage up to 50%. According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%. With enabled zstd at kafka topic, it shows no diffence in traffic. So it doesn't make much sense to use it. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225 * mention eb61a7dd68b834b08d01727a918f207700348ada at changelog * app/vmagent: bumps kafka lib version it allows compiling vmagent for arm64 machines fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271 * mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log * app/vmagent: adds natural concurrency for kafka consumer it should improve performance for data consumption https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957 * mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca * Update app/vmagent/kafka/consumer.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update app/vmagent/kafka/consumer_cgo.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-03-15 13:03:51 -07:00
Alexander Marshalov	3f10f19842	updated vars doc for operator (#3960 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-15 12:19:53 -07:00
dmitryk-dk	1ff82e23af	docs: cleanup	2023-03-15 12:19:16 -07:00
dmitryk-dk	e1c2c91bb6	app/vmctl: update managed quickstart guide	2023-03-15 12:18:48 -07:00
Aliaksandr Valialkin	5375e789fd	vendor: `make vendor-update`	2023-03-14 16:20:05 -07:00
Aliaksandr Valialkin	8be6c85eb3	vendor: update github.com/klauspost/compress from v1.16.0 to v1.16.3	2023-03-14 16:20:03 -07:00
Zakhar Bessarab	a3f364869f	doc: add guide for vmgateway configuration with OpenID and Grafana (#3951 ) docs: add guide for vmgateway configuration with OpenID and Grafana	2023-03-14 16:11:29 -07:00
Zakhar Bessarab	d1d108fe77	lib/storage: log original labels set when label value is truncated (#3952 ) lib/storage: log original labels set when label value is truncated	2023-03-14 16:11:02 -07:00
Dmytro Kozlov	235477628e	app/vmctl: integration test for native protocol (#3947 ) * app/vmctl: integration test for native protocol * app/vmctl: implemented two integration tests * app/vmctl: cleanup * app/vmctl: split storage init and filling data logic * app/vmctl: cleanup * app/vmctl: remove storage from server, used initialization process * app/vmctl: prepare for parallel run, code cleanup * app/vmctl: code cleanup * app/vmctl: remove unused field	2023-03-14 16:08:40 -07:00
Aliaksandr Valialkin	9aa6b63160	docs/Single-server-VictoriaMetrics.md: clarify that the cache directory can be removed manually when VictoriaMetrics is stopped	2023-03-13 00:24:01 -07:00
Aliaksandr Valialkin	d345619424	docs/CHANGELOG.md: document v1.87.3	2023-03-13 00:21:00 -07:00
Aliaksandr Valialkin	1ae73a5fe5	docs/CHANGELOG.md: document v1.79.11 LTS release	2023-03-12 23:23:05 -07:00
Aliaksandr Valialkin	4d81f1c235	deployment: update VictoriaMetrics release from v1.88.0 to v1.89.1 See https://docs.victoriametrics.com/CHANGELOG.html#v1891	2023-03-12 20:05:32 -07:00
Aliaksandr Valialkin	47c5b77c8c	docs/CHANGELOG.md: cut v1.89.1	2023-03-12 19:14:29 -07:00
Aliaksandr Valialkin	3b4a3583bc	app/vmselect/promql: prevent from `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade to v1.89.0	2023-03-12 19:09:11 -07:00
Aliaksandr Valialkin	a70b517d92	docs/CHANGELOG.md: cut v1.89.0	2023-03-12 17:29:50 -07:00
Aliaksandr Valialkin	cf7d8811f6	app/vmselect/vmui: `make vmui-update` after `00a0816ab1`	2023-03-12 17:22:28 -07:00
Yury Molodov	aebc441251	vmui: predefined dashboards docs (#3895 ) * fix: correct display predefined panels * docs: update the documentation for predefined dashboards	2023-03-12 17:22:27 -07:00
Aliaksandr Valialkin	7d37ab63a5	Makefile: update golangci-lint from v1.51.1 to v1.51.2 See https://github.com/golangci/golangci-lint/releases/tag/v1.51.2	2023-03-12 17:08:34 -07:00
Aliaksandr Valialkin	a6a4beb89a	app/vmselect: remove data race on updating EvalConfig.IsPartialResponse from concurrently running goroutines This properly returns `is_partial: true` for partial responses.	2023-03-12 16:53:03 -07:00
Artem Navoiev	0186c7fdfb	fix typos on image Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-12 16:32:37 -07:00
Aliaksandr Valialkin	5cd60c54d3	app/vmselect/promql: prevent from SIGBUS crash on architecures, which deny unaligned access to 8-byte words (e.g. ARM) Thanks to @oliverpool for nailing down the root cause of the issue and for the initial attempt to fix it at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927	2023-03-12 16:29:18 -07:00
Aliaksandr Valialkin	9cd0a2e875	vendor: `make vendor-update`	2023-03-12 03:14:09 -07:00
Yury Molodov	2a1bc14984	vmui: remove send step param for instant queries (#3931 ) * fix: remove step param for instant queries (#3896) * vmui: remove send step param for instant queries * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-12 03:14:05 -07:00
Aliaksandr Valialkin	6de1d034c8	docs/CHANGELOG.md: document `113a89904d`	2023-03-12 01:58:02 -08:00
Aliaksandr Valialkin	97479bd9e3	docs/CHANGELOG.md: yet another typo fix	2023-03-12 01:52:17 -08:00
Nikolay	113a89904d	lib/vmselectapi: fixes regression for disable compression setting (#3932 ) after vmselect api refactoring it wasn't possible to disable response cache. This patch restores correct behavior for rpc.disableCompression flag	2023-03-12 01:48:08 -08:00
Aliaksandr Valialkin	22e8bd7222	docs/CHANGELOG.md: typo fix	2023-03-12 01:04:23 -08:00
Aliaksandr Valialkin	e491fee1f4	app/vmselect/netstorage: do not intern string representation of MetricName for time series received from vmstorage It has been appeared that this interning may lead to increased memory usage and increased CPU usage when vmselect performs queries, which select big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863	2023-03-12 00:44:08 -08:00
Aliaksandr Valialkin	d106ababb8	docs/CHANGELOG.md: document `927d9da270`	2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin	094fb31089	app/vmctl/README.md: remove trailing space from the line added at `4c3bc04efa`	2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin	5ac8f89d2a	docs/CHANGELOG.md: typo fixes	2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin	868970bd9e	docs/CHANGELOG.md: document c9f44daaee8f4282d9ed41e3ba799c7a33841313	2023-03-12 00:27:33 -08:00
Roman Khavronenko	8b907fffdb	vmalert: support concurrent reading from object storage (#532 ) * vmalert: support concurrent reading from object storage Config reading from GCS or S3 can be slow if object storage contains a big number of files. Object storages are usually fast for downloading and are slow for individual operations. If there would be thousands of files to read, vmalert could spend significant time for retrieving those because it is done sequentially. The change introduces ability to read configs from object storage concurrently. By default, both GCS and S3 are now read with 50 concurrent readers. This significantly reduces the load time: * loading 500 files with concurrency=1 takes 27s * loading 500 files with concurrency=50 takes <1s * vmalert: add note to Changelog * vmalert: cleanup * vmalert: use ticker properly * app/vmalert: improve status reporting during config loading * vmalert: support concurrent reading from object storage Config reading from GCS or S3 can be slow if object storage contains a big number of files. Object storages are usually fast for downloading and are slow for individual operations. If there would be thousands of files to read, vmalert could spend significant time for retrieving those because it is done sequentially. The change introduces ability to read configs from object storage concurrently. By default, both GCS and S3 are now read with 50 concurrent readers. This significantly reduces the load time: * loading 500 files with concurrency=1 takes 27s * loading 500 files with concurrency=50 takes <1s * app/vmalert: make linter happy	2023-03-12 00:25:34 -08:00
Nikolay	3caf898a83	lib/storage: correctly handle io.EOF error for pre-fetched metrics (#3946 ) io.EOF shouldn't be returned from this function. It breaks all search API logic and may result in empty query results.	2023-03-12 00:19:58 -08:00
Alexander Marshalov	78e9e4742d	Stream aggregation doc improvements based on users feedback (#3934 ) docs: stream aggregation doc improvements based on users feedback	2023-03-12 00:19:22 -08:00
Roman Khavronenko	2b8edaa609	Dashboards upd (#3942 ) * dashboards/cluser: use `quantile` since `median` isn't supported by PromQL Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/: add `restarts` annotation to show when there were restarts The cluster's annotation query is aggregated `by job`, while vmagent/vmalert are aggregated `by job, instance`. This is because cluster dashboard can contains too many instances and annotation could become too noisy. Signed-off-by: hagen1778 <roman@victoriametrics.com> dashboards/*: support instance filter in Version annotation Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-12 00:14:32 -08:00
Zakhar Bessarab	3b7152b1d8	docs: add a note about cache reset for vmalert backfilling docs (#3940 ) docs: add a note about cache reset for vmalert backfilling docs	2023-03-12 00:13:00 -08:00
Dmytro Kozlov	4029c539b9	app/vmctl: update importing tips when migrating data with overlapping time range (#3941 ) app/vmctl: update importing tips when migrating data with overlapping time range	2023-03-12 00:10:08 -08:00
Dmytro Kozlov	f7d3efe3db	app/vmctl: add support of basic auth and barer token (#3921 ) app/vmctl: add support of basic auth and bearer token	2023-03-12 00:08:04 -08:00
Roman Khavronenko	310b380a03	app/vmalert: log number of configration files found for each specified `-rule` (#3936 ) The change also introduces `List` method to `FS` interface. The `List` method can be used for wildcard support in object storage FS. Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2023-03-11 23:40:40 -08:00
Dmytro Kozlov	afc6a6e218	app/vmctl: follow up after `09e3742a82` (#3937 ) app/vmctl: follow up after `09e3742a82`	2023-03-11 23:39:41 -08:00
Roman Khavronenko	b6ed977fa4	security: bump go version to 1.20.2 (#3935 ) upgrade Go builder from Go1.20.1 to Go1.20.2 See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved). Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-11 23:38:14 -08:00
Gowtam Lal	e2522baa9c	app/vmctl: Allow vmnative exports to skip HTTP keepalive. (#3909 ) app/vmctl: support HTTP keepalive disabling for vm-native mode	2023-03-11 23:34:45 -08:00
Denys Holius	e8a0d60ea5	Adds snap badge to README.md (#3930 ) docs: adds snap badge to README.md	2023-03-11 23:31:38 -08:00
Aliaksandr Valialkin	54fe207cc0	all: follow-up for `7a3e16e774` - Sync the description for -httpListenAddr.useProxyProtocol command-line flag at vmagent and vmauth, so it is consistent with the description at vmauth and victoria-metrics - Add a sample of panic text to docs/CHANGELOG.md, so it could be googled - Mention the -httpListenAddr.useProxyProtocol command-line flag in the description for the bugfix Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335	2023-03-08 01:42:58 -08:00
Aliaksandr Valialkin	bc24d35153	app/vmselect/vmui: `make vmui-update` after `bbf8e459a0`	2023-03-08 01:40:18 -08:00
Aliaksandr Valialkin	37981b276b	docs/CHANGELOG.md: improve description for `4b136abff8`	2023-03-08 01:39:59 -08:00
Aliaksandr Valialkin	a911889817	docs/CHANGELOG.md: improve the description of the bugfix at `62beea23f7` - Make the description easier to read by humans :) - Add a link to VictoriaMetrics datasource plugin for Grafana, so users could easily discover it	2023-03-08 01:39:39 -08:00
Aliaksandr Valialkin	bb5a39e5d1	docs/CHANGELOG.md: clarify the description for `6bfe9cc733` - Add the panic message to the description, so it is easier to google - Add a link to the corresponding bugreport Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897	2023-03-08 01:38:54 -08:00
Aliaksandr Valialkin	c6382d9538	app/vmctl/README.md: remove trailing space after `cc5b916237`	2023-03-08 01:38:27 -08:00
Aliaksandr Valialkin	e91fe74618	all: typo fixes of the same type as in the `d056be710b`	2023-03-08 01:37:28 -08:00
Aliaksandr Valialkin	15f5639eb4	docs/CHANGELOG.md: clarify the description for the change at `8bab50dc29` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600	2023-03-08 01:35:56 -08:00
Aliaksandr Valialkin	b60594a548	app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c - Document the fix - Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient() This simplifies the fix to the following diff: diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go index 099899c19..70b904af4 100644 --- a/app/vmagent/remotewrite/client.go +++ b/app/vmagent/remotewrite/client.go @@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq persiste } c.sendBlock = c.sendBlockHTTP - return c -} - -func (c client) init(argIdx, concurrency int, sanitizedURL string) { useVMProto := forceVMProto.GetOptionalArg(argIdx) usePromProto := forcePromProto.GetOptionalArg(argIdx) if useVMProto && usePromProto { @@ -173,6 +169,10 @@ func (c client) init(argIdx, concurrency int, sanitizedURL string) { } c.useVMProto = useVMProto + return c +} + +func (c client) init(argIdx, concurrency int, sanitizedURL string) {	2023-03-08 01:35:31 -08:00
Dmytro Kozlov	2cf6797a24	app/vmagent: fix panic if auth config not defined (#530 )	2023-03-08 01:35:04 -08:00
Aliaksandr Valialkin	eb62b5548d	docs/CHANGELOG.md: document `ec2abf9b69`	2023-03-08 01:34:08 -08:00
Alexander Marshalov	0b7f83c846	added documentation about new templates field of vmalertmanager specification in operator (https://github.com/VictoriaMetrics/operator/issues/592 ) (#3924 ) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-08 01:33:42 -08:00
Nikolay	88f10d24a0	lib/netutil: fixes panic at proxy protocol (#3905 ) it may occur if non proxy protocol message received by tcp server. Listener Accept method must return only non-recoverable errors. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335	2023-03-08 01:33:01 -08:00
Yury Molodov	3e3e23a483	vmui: fix display of selected value in the selector (#3919 ) vmui: fix selected value in dropdowns for Explore page	2023-03-08 01:13:54 -08:00
Nikolay	77e0e847a8	docs: updates operator api.md (#3922 )	2023-03-08 01:12:16 -08:00
Roman Khavronenko	fa3b2bd205	app/vmalert: do not wait for group start on removal (#3891 ) Each group in vmalert starts with an artifical delay to avoid thundering herd problem. For some groups with high evaluation intervals, the delay could be significant. If during this delay user will remove the group from the config and hot-reload it - vmalert will have to wait until the delay ends. This results into slow config reloading and UI hang. The change moves the start-delay logic back to the group's `start` method. Now, group can immediately exit from the delay when `group.close()` method is called. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 01:10:41 -08:00
Craig Rodrigues	4c94fe0f0a	docs: move installation methods further up in README (#3904 )	2023-03-08 01:09:35 -08:00
Dmytro Kozlov	d59a20c0a7	docs: follow up after `4b136abff8` (#3918 ) docs: follow up after `4b136abff8`	2023-03-08 01:08:07 -08:00
Alexander Marshalov	c682609fc4	added doc for placeholder support in vmagent specification for operator (https://github.com/VictoriaMetrics/operator/issues/592 ) (#3916 )	2023-03-08 01:07:14 -08:00
Gowtam Lal	0efdf47571	app/vmctl: Add ability to set headers for vm-native HTTP requests. (#3906 ) app/vmctl: Add ability to set headers for vm-native HTTP requests	2023-03-08 01:06:32 -08:00
Roman Khavronenko	e2527917a9	docs: follow-up after `62beea23f7` (#3907 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 01:01:16 -08:00
Roman Khavronenko	8d65c6e46e	github: fix validation errors (#3903 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:52:27 -08:00
Roman Khavronenko	b37ae04ae0	docs: follow-up `e781e22c9c` (#3902 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:51:34 -08:00
Roman Khavronenko	c160923ae1	github: add a Question issue type (#3901 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:50:42 -08:00
Yury Molodov	c383c9b864	vmui: show query error (#3890 ) * add links support with old query params * show error after execute query	2023-03-08 00:49:36 -08:00
Artem Makhortov	f9723aca44	doc: vmctl vm-native-step-interval supported values (#3899 )	2023-03-08 00:47:37 -08:00
Haleygo	b301455150	fix some typo (#3898 )	2023-03-08 00:32:57 -08:00
Artem Navoiev	950abf7457	remove image width for vmalert managed vm guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-08 00:26:53 -08:00
Artem Navoiev	0ba6c42ea2	change title Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-08 00:26:22 -08:00
Artem Navoiev	470c5561d2	add vmalert managed vm integration guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-03-08 00:25:37 -08:00
Dmytro Kozlov	ff0e63ef0d	app/vmctl: add backoff retries to native protocol (#3859 ) app/vmctl: vm-native - split migration on per-metric basis `vm-native` mode now splits the migration process on per-metric basis. This allows to migrate metrics one-by-one according to the specified filter. This change allows to retry export/import requests for a specific metric and provides a better understanding of the migration progress. --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:24:08 -08:00
Roman Khavronenko	b176247e16	vmalert: cancel in-flight requests on group's update or close (#3886 ) When group's update() or close() method is called, the group still need to wait for its current evaluation to finish. Sometimes, evaluation could take a significant amount of time which slows configuration update or vmalert's graceful shutdown. The change interrupts current evaluation in order to speed up the graceful shutdown or config update procedures. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:10:11 -08:00
Roman Khavronenko	c9ee4e5e3d	dashboards: account for indexdb size in Bytes-per-Point panel (#3884 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-08 00:08:41 -08:00
Roman Khavronenko	843c8cae44	deployment/docker: fix typo (#3883 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-03-07 23:35:58 -08:00
Dmytro Kozlov	08df754903	app/vmctl: Increase http request timeout made by remote read client, add importing tips (#3879 ) app/vmctl: Increase default http request timeout made by remote read client	2023-03-07 23:35:25 -08:00
Nikolay	361e1b1165	lib{mergset,storage}: prevent possible race condition with logging st… (#3900 ) (#3917 ) lib{mergset,storage}: prevent possible race condition with logging stats for merges Previously partwrapper could be release by background process and reference for part may be invalid during logging stats. It will lead to panic at vmstorage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897	2023-03-06 11:11:08 +01:00
Aliaksandr Valialkin	57f50ace96	.github/workflows/check-licenses.yml: use the correct version of Go - 1.20.1 - instead of 1.21.0	2023-02-27 19:25:23 -08:00
Aliaksandr Valialkin	223bb6f367	docs/CHANGELOG.md: document v1.79.10 release	2023-02-27 17:36:13 -08:00
Aliaksandr Valialkin	73514d04b0	vendor: `make vendor-update`	2023-02-27 15:33:02 -08:00
Aliaksandr Valialkin	ea1d7f28e5	docs/CHANGELOG.md: cut v1.88.1	2023-02-27 15:28:27 -08:00
Aliaksandr Valialkin	da419e7188	docs/CHANGELOG.md: link to the issue, which may benefit from `-internStringDisableCache` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692	2023-02-27 14:56:22 -08:00
Aliaksandr Valialkin	086a4b4fca	lib/bytesutil: add `-internStringDisableCache` and `-internStringCacheExpireDuration` command-line flags This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872	2023-02-27 14:18:02 -08:00
Aliaksandr Valialkin	1ad0d22e80	lib/storage: follow-up for `39cdc546dd` - Use flag.Duration instead of flagutil.Duration for -snapshotCreateTimeout, since the flagutil.Duration is intended mostly for big durations, e.g. days, months and years, while the -snapshotCreateTimeout is usually smaller than one hour. - Add links to https://docs.victoriametrics.com/#how-to-work-with-snapshots in docs/CHANGELOG.md, so readers could easily find the corresponding docs when reading the changelog. - Properly remove all the created directories on unsuccessful attempt to create snapshot in Storage.CreateSnapshot(). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551	2023-02-27 13:11:10 -08:00
Zakhar Bessarab	26682e369e	lib/storage: enhancements for snapshots process (#3873 ) * lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858) * lib/{mergeset,storage}: add timeout configuration for snapshots creation, remove incomplete snapshots from storage * docs: fix formatting * app/vmstorage: add metrics to track status of snapshots * app/vmstorage: use `vm_http_requests_total` metric for snapshot endpoints metrics, rename new flag to make name more clear Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmstorage: update flag name in docs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmstorage: reflect new metrics names change in docs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-27 13:11:06 -08:00
Zakhar Bessarab	1db010797e	lib/promscrape: correctly register `vm_promscrape_config_` metrics (#3876 ) lib/promscrape: set `vm_promscrape_config_last_reload_successful` to 1 if there was no promscrape config provided Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape: register `vm_promscrape_config_*` metrics only in case promscrape config is used Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-27 12:06:49 -08:00
Zakhar Bessarab	9d04bb6004	doc: add changelog reference for vmgateway OpenID discovery (#3877 ) * doc: add changelog reference for vmgateway OpenID discovery * doc: add vmgateway docs --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-27 12:06:48 -08:00
Aliaksandr Valialkin	d3605ad072	app/vmselect/promql: fix panic when calculating `aggr_func(rollup*())` The panic has been introduced in `dac21d874b`	2023-02-27 11:48:38 -08:00
Zakhar Bessarab	215206bded	app/vmgateway: add new flag doc Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-27 11:18:37 -08:00
Zakhar Bessarab	08a61d5c13	app/vmgateway: fix typo in docs Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-27 11:18:11 -08:00
Zakhar Bessarab	1b4871a9da	app/vmgateway: add OpenID discovery of JWKS endpoints Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-27 11:17:44 -08:00
Aliaksandr Valialkin	d9581cf3f3	app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol	2023-02-27 11:04:11 -08:00
Aliaksandr Valialkin	06ac40aafa	lib/httpserver: use github.com/klauspost/compress/gzhttp for compressing http responses This allows removing gzip-related code from lib/httpserver.	2023-02-27 10:35:26 -08:00
Dmytro Kozlov	65d9e281d6	app/vmctl: skip series if measurement not found (#3869 ) app/vmctl: skip measurements with no fields for influxdb mode	2023-02-27 10:35:13 -08:00
Dmytro Kozlov	afa3678b01	app/vmctl: enable version flag (#3868 )	2023-02-27 10:34:26 -08:00
Aliaksandr Valialkin	bbd5914eb1	all: add makefile rules for GOARCH=s390x for all the VictoriaMetrics components This is a follow-up for `007530f882`	2023-02-26 12:38:48 -08:00
Aliaksandr Valialkin	18dd0d1dbf	.golangci.yml: properly enable `revive` linter and fix all the warnings it detects	2023-02-26 12:19:58 -08:00
Aliaksandr Valialkin	1e156ac3c3	app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-26 12:19:53 -08:00
Aliaksandr Valialkin	be925727c7	deployment/marketplace: update VictoriaMetrics release from v1.87.1 to v1.88.0	2023-02-24 18:59:02 -08:00
Aliaksandr Valialkin	1f24771779	deployment/docker: update VictoriaMetrics docker tag from v1.87.1 to v1.88.0	2023-02-24 18:59:01 -08:00
Aliaksandr Valialkin	a5e79b6f38	docs/CHANGELOG.md: cut v1.88.0	2023-02-24 17:54:35 -08:00
Aliaksandr Valialkin	328eb3cf86	vendor: `make vendor-update`	2023-02-24 17:27:04 -08:00
Aliaksandr Valialkin	90114759fa	docs: update -help output after `e1c3267e34`	2023-02-24 17:16:24 -08:00
Roman Khavronenko	66d0b45651	vmselect/promql: check for deadline in `count_values` fn (#3806 ) * vmselect/promql: check for deadline in `count_values` fn `count_values` could be very slow during the data processing. Checking for deadline between iterations supposed to reduce probability of exceeding `search.maxQueryDuration`. The change also adds a new trace record, which captures the time spent in aggregation function. Before that, the trace for aggr funcs could be confusing since it doesn't account for all the places where time was spent. Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-24 17:10:38 -08:00
Aliaksandr Valialkin	33df81cb4b	docs/CHANGELOG.md: document v1.87.2 release	2023-02-24 16:14:41 -08:00
Aliaksandr Valialkin	698aa1dd2a	docs/CHANGELOG.md: document v1.79.9 release	2023-02-24 15:10:58 -08:00
Roman Khavronenko	79eb33556e	metricsql: support optional 2nd argument for rollup functions (#3841 ) * metricsql: support optional 2nd argument for rollup functions Support optional 2nd argument `min`, `max` or `avg` for rollup functions: * rollup * rollup_delta * rollup_deriv * rollup_increase * rollup_rate * rollup_scrape_interval If second argument is passed, then rollup function will return only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, `rollup_rate(requests_total[5m], "max")`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-24 13:48:30 -08:00
Aliaksandr Valialkin	21fb56a498	docs/CHANGELOG.md: document `d8eaa511b0`	2023-02-24 12:43:45 -08:00
Zakhar Bessarab	75b8733e0b	lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858 ) (#3867 )	2023-02-24 12:43:43 -08:00
Aliaksandr Valialkin	0a9f5e178e	docs/CHANGELOG.md: typo fix: `scrape scrape` -> `scrape`	2023-02-24 12:33:37 -08:00
Aliaksandr Valialkin	aed2dbe45e	lib/promscrape: follow-up for `43e104a83f` - Return immediately on context cancel during the backoff sleep. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747 - Add a comment describing why the second attempt to obtain the response from remote side is perfromed immediately after the first attempt. - Remove fasthttp dependency from lib/promscrape/discoveryutils - Set context deadline before calling doRequestWithPossibleRetry(). This simplifies the doRequestWithPossibleRetry() a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293	2023-02-24 12:25:36 -08:00
Zakhar Bessarab	5ea6d71cb3	fix: do not use exponential backoff for first retry of scrape request (#3824 ) * fix: do not use exponential backoff for first retry of scrape request (#3293) * lib/promscrape: refactor `doRequestWithPossibleRetry` backoff to simplify logic Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update lib/promscrape/client.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * lib/promscrape: refactor `doRequestWithPossibleRetry` to make it more straightforward Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-02-24 12:25:35 -08:00
Aliaksandr Valialkin	8efa9159cf	app/vmselect/promql: measure the time required for calculating the aggregate function from the prepared source time series	2023-02-23 20:06:02 -08:00
Aliaksandr Valialkin	dce8afa4c4	app/vmselect/vmui: `make vmui-update` after `d4fc0ed874`	2023-02-23 19:26:08 -08:00
Yury Molodov	57c257810d	vmui: improve mobile ui (#3848 ) * feat: improve mobile ui * feat: improve mobile ui * fix: change style server url * fix: improve ExploreMetrics mobile * fix: display global settings on all pages	2023-02-23 19:26:01 -08:00
Aliaksandr Valialkin	3563c6882e	docs: update --help descriptions after recent changes	2023-02-23 19:02:32 -08:00
Aliaksandr Valialkin	6369c88a68	app/vmselect: add -search.logQueryMemoryUsage command-line flag for logging queries, which take big amounts of memory Thanks to @michal-kralik for initial attempts for this feature: - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3651 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3715 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553	2023-02-23 18:52:44 -08:00
Aliaksandr Valialkin	04365b949e	lib/protoparser: fix golangci-lint warning after `f579cac297`	2023-02-23 18:50:00 -08:00
Aliaksandr Valialkin	f579cac297	app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag, which can be used for forcing Prometheus remote write protocol in cases when the remote storage supports VictoriaMetrics remote write protocol. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-23 17:38:47 -08:00
Aliaksandr Valialkin	bb5a3dc153	lib/promscrape/discovery/kuma: substitute blocking HTTP call with non-blocking HTTP call at discoveryutils.Client	2023-02-23 15:14:00 -08:00
Denys Holius	c708ce1985	Fix some typos and adds improvements for packer builds (#3825 ) * update helper scripts to latest versions * added missed command for initialisation variables from Makefile * deployment/marketplace/vultr/helper-scripts/vultr-helper.sh: update helper script to latest version * fixed typo for using VM_VERSION variable * added an example of specifying the VM_VERSION and tokens for API's * set packer logging to STDOUT by default	2023-02-23 14:54:55 -08:00
Aliaksandr Valialkin	e299854077	docs/CHANGELOG.md: document `6d019a3c37` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830	2023-02-22 19:24:33 -08:00
Mattias Ängehov	3904b8959e	Azure Service Discovery - Fix token fetch for Container Apps/App Services (#3832 ) * Modify API version when running in Container App * Handle expires on from token response Response from IMDS does not always contain expires in value which is currently used to get the token expiry time. An example resources that doesn't provide it are Container Apps and App Service. Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com> * Fix client id parameter for user assigned identity * Apply suggestions from code review --------- Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com> Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2023-02-22 19:24:23 -08:00
Aliaksandr Valialkin	0c60e4a30a	all: consistently use http.Method{Get,Post,Put} across the codebase This is a follow-up after `9dec3c8f80`	2023-02-22 19:01:09 -08:00
my-git9	7d86c5c94a	chore: Use http constants to replace numbers (#3846 ) Signed-off-by: xin.li <xin.li@daocloud.io>	2023-02-22 18:59:32 -08:00
Alexander Marshalov	9bb9bd266b	fix interpolate function for filling only intermediate gaps (#3816 ) (#3857 ) * fix interpolate function for filling only intermediate gaps (#3816) * Update docs/CHANGELOG.md --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-22 18:41:29 -08:00
Yury Molodov	ecdbcb16ed	fix: change query params update (#3860 )	2023-02-22 18:26:33 -08:00
Alexander Marshalov	66764ea4a0	fixed typo in dns+srv documentation (#3861 )	2023-02-22 17:52:25 -08:00
Aliaksandr Valialkin	9609a44cc7	docs/CHANGELOG.md: document `d2b92d3264` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747	2023-02-22 17:50:55 -08:00
Aliaksandr Valialkin	1b70238dca	lib/promscrape/discovery/kuma: follow-up for `317fef95f9` - Do not generate __meta_server label, since it is unavailable in Prometheus. - Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md, so users could click it and read the docs without the need to search the corresponding docs. - Remove kumaTarget struct, since it is easier generating labels for discovered targets directly from the response returned by Kuma. This simplifies the code. - Store the generated labels for discovered targets inside atomic.Value. This allows reading them from concurrent goroutines without the need to use mutex. - Use synchronouse requests to Kuma instead of long polling, since there is a little sense in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval. - Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used. - Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus. - Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability. - Remove unused fields from discoveryRequest and discoveryResponse. - Update tests. - Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config. - Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback, since these are public types. Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label, since usually this label is set by the Prometheus itself to __address__ after the relabeling phase. See https://www.robustperception.io/life-of-a-label/ Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389 See https://github.com/prometheus/prometheus/issues/7919 and https://github.com/prometheus/prometheus/pull/8844 as a reference implementation in Prometheus	2023-02-22 17:50:54 -08:00
Aliaksandr Valialkin	b7d13c3478	lib/promscrape/discovery: add a comment explaining why duplicates are removed from the generated target labels	2023-02-22 17:50:42 -08:00
Aliaksandr Valialkin	55708fa474	docs/CHANGELOG.md: document `110c3896e7` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600	2023-02-22 17:50:13 -08:00
Aliaksandr Valialkin	b3aed1aebc	app/vmagent/remotewrite: removed unneeded code in testPushWriteRequest after `57801660ab`	2023-02-22 17:50:03 -08:00
Zakhar Bessarab	2c05066f19	lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3853 ) * lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (see #3747) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape: fix order of params for `doRequestWithPossibleRetry` to follow codestyle Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape: accept deadline explicitly and extend passed context for local use Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-22 17:49:43 -08:00
Artem Navoiev	3abd6b367e	Add Dig Security case study Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-02-22 17:42:51 -08:00
Alexander Marshalov	173643a771	add kuma_sd_config for Kuma Control Plane targets discovery (#3389 ) (#3840 )	2023-02-22 17:41:43 -08:00
Dmytro Kozlov	88d2a6c3a3	app/vmctl: add retry backoff policy (#3844 ) app/vmctl: move retries logic into a separate pkg	2023-02-22 12:17:21 -08:00
Alexander Marshalov	29fa78a310	Fix TestPushWriteRequest for remotewriteprotocol for pure-test (with native zstd) (#3850 ) app/vmagent: fix TestPushWriteRequest for pure-test (with native zstd) Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-02-22 12:16:29 -08:00
Artem Navoiev	522179fbde	docs: fix typo OpentTSDB -> OpenTSDB (#3854 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-02-22 12:11:48 -08:00
Aliaksandr Valialkin	ff8c57a964	app/vmselect: allow zero value for `-search.latencyOffset` command-line flag See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218	2023-02-21 18:07:27 -08:00
Aliaksandr Valialkin	d409d3cadb	vendor: `make vendor-update`	2023-02-21 18:07:26 -08:00
Aliaksandr Valialkin	9d6eeb0c07	go.mod: update github.com/VictoriaMetrics/fastcache from v1.12.0 to v1.12.1	2023-02-21 17:51:47 -08:00
Aliaksandr Valialkin	013e2bacaf	vendor: update github.com/VictoriaMetrics/fasthttp from v1.1.0 to v1.2.0 The v1.2.0 adds HostClient.DoCtx() function, which is needed by https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747 for implementing fast canceling of pending requests to scrape targets on config update	2023-02-21 17:50:25 -08:00
Aliaksandr Valialkin	57de6ad5dd	docs/Articles.md: mention `rules backfilling via vmalert` article This is a follow-up for `5446ce0018`	2023-02-21 17:44:45 -08:00
Roman Khavronenko	a29c1d3a02	docs: mention rules replay blogpost in vmalert docs (#3851 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-21 17:44:32 -08:00
panguicai	1b75d66bcd	docs: update operator release name to be consistent with the following (#3845 ) Signed-off-by: panguicai008 <1121906548@qq.com>	2023-02-21 17:42:39 -08:00
Aliaksandr Valialkin	f783237866	docs/CHANGELOG.md: fix a link to VictoriaMetrics remote write protocol	2023-02-20 20:00:00 -08:00
Aliaksandr Valialkin	09b21fd800	docs/Single-server-VictoriaMetrics.md: remove `+` from `start` and `end` query args, since curl substitutes them with whitespace and breaks the query	2023-02-20 19:32:39 -08:00
Aliaksandr Valialkin	04ce4c4762	docs/guides/migrate-from-influx.md: remove misleading doublequotes from the query example	2023-02-20 19:25:07 -08:00
Zakhar Bessarab	8a35dc608f	vmgateway: add support of JWKS endpoint usage for JWT keys verification (#521 )	2023-02-20 19:23:00 -08:00
Roman Khavronenko	fd139b463b	docs: update vmalert docs (#3843 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-20 19:13:00 -08:00
Corporte Gadfly	7d99b19478	docs: typo fix (#3839 )	2023-02-20 19:11:59 -08:00
Aliaksandr Valialkin	fc5ea31a38	docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol The 10x savings are reproduced only on artificial data. The savings on production data are usually in the range 2x-4x.	2023-02-20 19:11:11 -08:00
Aliaksandr Valialkin	bf359e8cc0	docs/vmagent.md: mention that Mimir doesnt support backfilling	2023-02-20 19:06:56 -08:00
Aliaksandr Valialkin	80c6d1e24c	app/vmagent: add support for VictoriaMetrics remote write protocol, which allows saving up to 10x on network bandwidth costs under high load Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-20 18:40:40 -08:00
Aliaksandr Valialkin	a15da5ff73	app/vmselect/promql: add `share(q)` aggregate function for normalizing results across multiple time series in [0..1] value range per each timestamp and aggregation group	2023-02-18 22:43:54 -08:00
Aliaksandr Valialkin	84b5532bc1	app/vmselect/promql: add range_zscore(q) and range_trim_zscore(z, q) functions These functions may be useful for dropping outliers at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759	2023-02-18 22:43:53 -08:00
Aliaksandr Valialkin	94334ed0bb	vendor: `make vendor-update`	2023-02-18 15:37:05 -08:00
Aliaksandr Valialkin	58779363b4	app/vmalert/README.md: sync with docs/vmalert.md after `6ef6f3a771`	2023-02-18 15:21:31 -08:00
Haleygo	9a274567f1	vmalert: fix maxResolveDuration flag note (#3827 ) Signed-off-by: Haleygo <hui.wang@daocloud.io>	2023-02-18 15:20:30 -08:00
Duc Tran	7fe3cd3110	docs: fix links to alerts and alertmanager (#3829 )	2023-02-18 15:19:48 -08:00
Aliaksandr Valialkin	450b6f6d39	app/vmselect/promql: add range_mad(q) and range_trim_outliers(k, q) functions These functions may help trimming outliers during query time for the use case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759	2023-02-18 15:18:47 -08:00
Aliaksandr Valialkin	b572512dbd	vendor: update github.com/valyala/gozstd from v1.17.0 to v1.18.0	2023-02-18 13:55:27 -08:00
Aliaksandr Valialkin	7274424252	app/vmui: tooltip formatting enhancements according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706#issuecomment-1429980038	2023-02-14 23:38:05 -08:00
Aliaksandr Valialkin	ecc84f7923	all: update Go builder from Go1.20.0 to Go1.20.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved	2023-02-14 23:05:54 -08:00
Aliaksandr Valialkin	d63862b3ab	vendor: `make vendor-update`	2023-02-13 11:14:30 -08:00
Aliaksandr Valialkin	9fd003d54a	all: rename ParseStream -> stream.Parse This is a follow-up for `057698f7fb`	2023-02-13 10:53:12 -08:00
Aliaksandr Valialkin	f987fb9c8b	lib/protoparser/promremotewrite: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:48:11 -08:00
Aliaksandr Valialkin	c54d17b006	lib/protoparser/native: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:44:27 -08:00
Aliaksandr Valialkin	086516a02b	lib/protoparser/clusternative: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:38:02 -08:00
Aliaksandr Valialkin	75cf5a8939	lib/protoparser/graphite: extract stream parsing code into a separate stream package	2023-02-13 10:33:24 -08:00
Aliaksandr Valialkin	1801fa6c5c	lib/protoparser/csvimport: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:26:29 -08:00
Aliaksandr Valialkin	41feed813d	lib/protoparser/vmimport: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:22:00 -08:00
Aliaksandr Valialkin	66f0a78810	lib/protoparser/opentsdbhttp: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:15:15 -08:00
Aliaksandr Valialkin	67c0281535	lib/protoparser/opentsdb: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:04:14 -08:00
Aliaksandr Valialkin	1add6c3fa0	lib/protoparser/influx: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 09:59:56 -08:00
Aliaksandr Valialkin	b691d02b92	lib/protoparser/datadog: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 09:53:20 -08:00
Roman Khavronenko	867b7e5688	lib/protoparser/prometheus: move `streamparser` to subpackage (#3814 ) `lib/protoparser/prometheus` is used by various applications, such as `app/vmalert`. The recent change to the `lib/protoparser/prometheus` package introduced a new dependency of `lib/writeconcurrencylimiter` which exposes some metrics. Because of the dependency, now all applications which have this dependency also expose these metrics. Creating a new `lib/protoparser/prometheus/stream` package helps to remove these metrics from apps which use `lib/protoparser/prometheus` as dependency. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-13 09:44:47 -08:00
Roman Khavronenko	c6251ec8aa	docs: improve troubleshooting docs for vmalert (#3812 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-13 09:42:18 -08:00
Aliaksandr Valialkin	caa2497aca	docs/CHANGELOG.md: improve the docs for `8ea02eaa8e`	2023-02-13 09:41:22 -08:00
Roman Khavronenko	5941b5579d	follow-up after `d1cbc35cf6` (#3813 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-13 09:37:09 -08:00
Droxenator	3961836476	fixed opentsdbListenAddr timestamp conversion (#3810 ) Co-authored-by: Andrei Ivanov <a.ivanov@corp.mail.ru>	2023-02-13 09:35:23 -08:00
Artem Navoiev	253a78b9ab	change docs for VictoriaMetrics Managed (#3803 ) * change docs for VictoriaMetrics Managedd Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * Update docs/managed-victoriametrics/user-managment.md Co-authored-by: Max Golionko <8kirk8@gmail.com> --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Max Golionko <8kirk8@gmail.com>	2023-02-13 09:33:32 -08:00
Oleksandr Redko	0e1c395609	app,lib: fix typos in comments (#3804 )	2023-02-13 09:32:35 -08:00
Aliaksandr Valialkin	a02576349a	app/vmauth: allow specifying `max_concurrent_requests` value on a per-user basis bigger than the `-maxConcurrentPerUserRequests` value	2023-02-11 20:53:20 -08:00
Aliaksandr Valialkin	8355e4a1ee	docs/sd_configs.md: properly escape __address__ string	2023-02-11 14:46:02 -08:00
Aliaksandr Valialkin	c8e41100e6	docs/sd_configs.md: document how the __address__ label is generated per each discovered target	2023-02-11 14:42:10 -08:00
Aliaksandr Valialkin	e6616c74a2	lib/promscrape/discovery/openstack: use port 80 for the discovered target by default if it isnt specified in the config	2023-02-11 14:42:09 -08:00
Aliaksandr Valialkin	27b3209816	app/vmui: show `median` instead of `avg` on graph tooltip and line legend Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706	2023-02-11 12:52:50 -08:00
Aliaksandr Valialkin	9053745a6f	lib/{mergeset,storage}: allow at least 3 concurrent flushes during background merges on systems with 1 or 2 CPU cores This should prevent from data ingestion slowdown and query performance degradation on systems with small number of CPU cores (1 or 2), when big merge is performed. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2023-02-11 12:09:13 -08:00
Roman Khavronenko	381dce79e6	dashboards: use `median` instead of `avg` (#3800 ) `avg` can be affected by just one outlier, which may lead to false conclusions. `median` is supposed to reflect reality better by leveling outliers out. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-11 12:09:09 -08:00
Aliaksandr Valialkin	1a6d266d1a	docs/Articles.md: add a link to https://www.techetio.com/2022/08/21/evaluating-backend-options-for-prometheus-metrics/	2023-02-11 11:37:06 -08:00
Aliaksandr Valialkin	db7f237da9	app/vmselect/promql: add `mad_over_time(m[d])` function See https://github.com/prometheus/prometheus/issues/5514	2023-02-11 01:06:39 -08:00
Aliaksandr Valialkin	4d9aaff3e1	all: update alpine base docker image from 1.17.1 to 1.17.2 See https://alpinelinux.org/posts/Alpine-3.17.2-released.html	2023-02-11 00:37:28 -08:00
Aliaksandr Valialkin	9ed5b872df	app/vmauth: improve load balancing by sending incoming requests to backends with the lowest number of concurrent requests While at it, stop sending requests to unavailable backend for 3 seconds before the next attempt. This should reduce the amounts of useless work and the number of useless network packets when the backend is temporarily unavailable.	2023-02-11 00:32:56 -08:00
Aliaksandr Valialkin	6c6b9c3d93	app/vmauth: add `-maxConcurrentPerUserRequests` command-line option for limiting the number of concurrent requests on a per-user basis Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346	2023-02-10 21:58:30 -08:00
Aliaksandr Valialkin	769d7da25a	app/vmauth: automatically retry failing GET requests on the remaining backends	2023-02-09 21:06:09 -08:00
Dmytro Kozlov	c80fc8c77f	app/vmauth: add concurrent requests limit per auth record (#3749 ) * app/vmauth: add concurent requests limit per auth record * app/vmauth: added clarification comment * app/vmauth: remove unused code * app/vmauth: move read from limiter * app/vmauth: fix text * app/vmauth: fix comments * - Clarify the docs for the max_concurrent_requests option at docs/vmauth.md - Clarify the description of the change at docs/CHANGELOG.md - Make sure that the -maxConcurrentRequests takes precedence over per-user max_concurrent_requests - Update tests for verifying that the max_concurrent_requests option is parsed properly Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-09 20:03:47 -08:00
Aliaksandr Valialkin	ca61c276ca	app/vmalert: follow-up after d3c64aae8768d58781ee7e358bd7f3d8e0eb836d - Document the change at docs/CHANGELOG.md - Add `Reading rules from object storage` section to docs/vmalert.md - Add `s3` prefix to command-line flags related to the configuration of s3 and gcs clients - Explicitly mention that reading rules from object storage is supported only in enterprise version	2023-02-09 19:10:36 -08:00
Roman Khavronenko	2eb9ca1889	vmalert: support object storage for rules (#519 ) * vmalert: support object storage for rules Support loading of alerting and recording rules from object storages `gcs://`, `gs://`, `s3://`. * review fixes	2023-02-09 19:10:34 -08:00
Aliaksandr Valialkin	d63a244895	deployment/docker: update VictoriaMetrics Docker images from v1.87.0 to v1.87.1	2023-02-09 15:53:36 -08:00
Aliaksandr Valialkin	a68bb879d4	vendor: `make vendor-update`	2023-02-09 14:48:38 -08:00
Aliaksandr Valialkin	6ee15f94f8	docs/CHANGELOG.md: document d621d50d4fb3b43a0bcb4419bee979f0192d38fe	2023-02-09 14:40:15 -08:00
Aliaksandr Valialkin	d02146663b	all: skip issues with low severity at `docker scan`	2023-02-09 14:25:33 -08:00
Aliaksandr Valialkin	34379d4cf1	all: run `apk update && apk upgrade` in base Alpine Docker image in order to get all the recent security fixes	2023-02-09 14:03:02 -08:00
Aliaksandr Valialkin	da9d1fea10	docs/CHANGELOG.md: cut v1.87.1 and mark 1.87.x as LTS release	2023-02-09 11:21:04 -08:00
Zakhar Bessarab	bbf663bd04	lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3791 ) * lib/promscrape: fix cancelling in-flight scrape requests during configuration reload when using `streamParse` mode (see #3747) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/CHANGELOG.md --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-09 11:18:36 -08:00
Aliaksandr Valialkin	82b5fa2fd0	app/vmui: UX enhancements for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706 - Display `min` value additionally to `avg`, `max` and `last` - Allow copy-n-pasting metric name with its labels from both legend and tooltup	2023-02-09 11:05:57 -08:00
Aliaksandr Valialkin	dc7b92bc34	docs/vmagent.md: clarify that automatically generated metrics contain all the target-specific labels, including `instance` and `job`	2023-02-09 11:05:57 -08:00
Air	fe6e1015fe	Possibly spelling in the Quick start	2023-02-09 11:05:57 -08:00
Yury Molodov	13f55fc21e	vmui: lazy loading predefined panels (#3795 ) * fix: change logic lazy loading predefined panels * app/vmselect/vmui: `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-09 00:12:29 -08:00
Yury Molodov	5a7c63a324	vmui: improve tenant selector (#3794 ) * fix: change styles tenant selector (#3792) * docs/CHANGELOG.md: document the change Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-09 00:09:36 -08:00
Aliaksandr Valialkin	cd78396adb	docs/CHANGELOG.md: document changes at v1.79.8 LTS release	2023-02-08 23:39:01 -08:00
Yury Molodov	54bfd22ec5	vmui: add last/max/avg values (#3789 ) * feat: add last/max/avg values (#3706) * fix: change filter exclude values * app/vmui: wip - improve the visualization for avg/max/last values - make getAvgFromArray() function resilient against inf/undefined/nil - export getLastFromArray() function, which is resilient against inf/undefined/nil - run `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-08 22:42:05 -08:00
Aliaksandr Valialkin	cfca7075c4	docs/CHANGELOG.md: document `75bcf86a31` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3740	2023-02-08 11:24:29 -08:00
Yury Molodov	0454f9489c	fix: turn off the local dashboards(#3740 ) (#3793 )	2023-02-08 11:24:28 -08:00
Aliaksandr Valialkin	8d359b549e	docs/CHANGELOG.md: add more context to the bugfix description in Nomad service discovery See `146fd2eca3`	2023-02-08 09:24:51 -08:00
Aliaksandr Valialkin	146b3bd088	lib/backup/azremote: fix after upgrading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from v0.6.1 to v1.0.0	2023-02-08 09:19:10 -08:00
Aliaksandr Valialkin	c7a4751c70	vendor: `make vendor-update`	2023-02-08 08:55:53 -08:00
Aliaksandr Valialkin	b3bfca16de	docs/CHANGELOG.md: document the change at `67b01329a0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761	2023-02-08 08:46:21 -08:00
Roman Khavronenko	4e922eb93b	Vmalert fixes (#3788 ) * vmalert: use group's ID in UI to avoid collisions Identical group names are allowed. So we should used IDs for various groupings and aggregations in UI. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: prevent disabling state updates tracking The minimum number of update states to track is now set to 1. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: properly update `debug` and `update_entries_limit` params on hot-reload Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: display `debug` field for rule in UI Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: exclude `updates` field from json marhsaling This field isn't correctly marshaled right now. And implementing the correct marshaling for it doesn't seem right, since json representation is mostly used by systems like Grafana. And Grafana doesn't expect this field to be present. Signed-off-by: hagen1778 <roman@victoriametrics.com> * fix test for disabled state Signed-off-by: hagen1778 <roman@victoriametrics.com> * fix test for disabled state Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-08 08:45:25 -08:00
Max Golionko	9d658ccce3	bump go to 1.20 in ci jobs (#3787 )	2023-02-08 08:43:37 -08:00
Roman Khavronenko	d260b90445	docs: follow-up after `2e4bfcce63` (#3785 ) `2e4bfcce63` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-08 08:37:48 -08:00
Karan Sharma	004a24c950	sd/nomad: panic in nomad watcher because of nil map (#3784 ) properly initialize url.Values	2023-02-08 08:37:02 -08:00
Aliaksandr Valialkin	f5595233c2	lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting `vm_concurrent_insert_capacity` and `vm_concurrent_insert_current` metrics This will result in proper calculations for the the alerting rule: avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761	2023-02-07 11:08:39 -08:00
Aliaksandr Valialkin	6149e4c319	Makefile: update golangci-lint from v1.50.1 to v1.51.1	2023-02-07 11:08:38 -08:00
earthgecko	e8924ebfba	Clarifications between standalone/cluster ingestion endpoints (#3771 ) docs: clarifications between standalone/cluster ingestion endpoints This is an attempt to make it a bit clearer to the user that the cluster version ingestion URLs are different from the standalone ones. I have also changed the order of the list items to make it a bit clearer and hopefully stop the user simply inferring that `/prometheus/api/v1` is only related to Prometheus data.	2023-02-07 09:34:02 -08:00
Max Golionko	41bf9a481a	CI: speedup build by 2.4x. restore nightly build (#3772 ) * setup docker buildx * add snyk integration * add go cache for docker build * cancel redundant job if there is new commit into same PR or branch	2023-02-07 09:31:00 -08:00
Max Golionko	eaa3716507	docs: update formatiing for k8s monitoring with Managed VictoriaMetrics (#3768 ) * jekyll formatting madness	2023-02-07 09:29:34 -08:00
Roman Khavronenko	80bf0bcf8c	vmalert: update docs (#3770 ) vmalert: update flags description Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-07 09:28:59 -08:00
Denys Holius	aa8411501c	fixed wrong vmstorage port number (#3769 )	2023-02-07 09:27:28 -08:00
Aliaksandr Valialkin	d8a54ab839	docs/CHANGELOG.md: fix formatting fro the change from `6fd10e8871`	2023-02-07 09:26:38 -08:00
Roman Khavronenko	96db7ac52c	vmalert: speed up state restore procedure on start (#3758 ) * vmalert: speed up state restore procedure on start Alerts state restore procedure has been changed to become asynchronous. It doesn't block groups start anymore which significantly improves vmalert's startup time. Instead, state restore is called by each group in their goroutines after the first rules evaluation. While previously state restore attempt was made for all loaded alerting rules, now it is called only for alerts which became active after the first evaluation. This reduces the amount of API calls to the configured remote read URL. This also means that `remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now and will have no effect if configured. See relevant issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2608 Signed-off-by: hagen1778 <roman@victoriametrics.com> * make lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-03 19:46:41 -08:00
Aliaksandr Valialkin	c9a32ebaf7	app/vmselect/vmui: `make vmui-update` after `e4c04b6dbe`	2023-02-03 19:34:20 -08:00
Yury Molodov	ccdf388094	vmui: set light theme for app mode (#3748 ) * fix: set light theme for app mode * fix: check inputTenantID flag * fix: rename inputTenantID to useTenantID	2023-02-03 19:34:18 -08:00
Aliaksandr Valialkin	26614da548	docs/CHANGELOG.md: document `f63f487787` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707	2023-02-03 19:30:40 -08:00
Yury Molodov	05873dee89	vmui: mobile view (#3742 ) * feat: add detect the system theme * fix: change logic fetch tenants * feat: add docs and info to cardinality page * feat: add mobile view #3707	2023-02-03 19:30:38 -08:00
Aliaksandr Valialkin	3db8d7cb01	dashboards: typo fix `Datapoints scanned per series` -> `Datapoints scanned per query`	2023-02-03 19:12:42 -08:00
Aliaksandr Valialkin	1985b73cd0	deployment/docker: update Go builder from Go1.19.5 to Go1.20.0 See https://go.dev/blog/go1.20	2023-02-03 15:05:48 -08:00
Aliaksandr Valialkin	6910dd2a98	docs/MetricsQL.md: add links to "rollup results" explanation	2023-02-03 11:10:49 -08:00
Aliaksandr Valialkin	06c3e303ec	docs/CHANGELOG.md: remove duplicate line after `b7cc0bfe3c`	2023-02-03 10:48:54 -08:00
Aliaksandr Valialkin	ee800d6336	docs/managed-victoriametrics/how-to-monitor-k8s.md: rename image files according to docs/assets/README.md	2023-02-03 10:46:49 -08:00
Max Golionko	2d419021dc	docs: move managed victoria metics guide into right folder (#3750 ) * move guide folder * image width control	2023-02-03 10:46:26 -08:00
Max Golionko	1d528f3ed6	Update docs/managed_victoriametrics/how-to-monitor-k8s.md Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-03 10:45:46 -08:00
Max Golionko	76d339b886	add guide to list of guides	2023-02-03 10:45:01 -08:00
Max Golionko	32b283d9c3	added k8s guide for managed VM	2023-02-03 10:43:45 -08:00
Max Golionko	2aa94191a0	disable codeql for docs. merge build and test back to one job (#3746 )	2023-02-02 08:37:23 -08:00
Aliaksandr Valialkin	f737674912	docs/CHANGELOG.md: typo fixes	2023-02-01 20:40:55 -08:00
Aliaksandr Valialkin	2cdc3fd500	deployment/docker: update VictoriaMetrics docker image tag from v1.86.2 to v1.87.0	2023-02-01 20:03:22 -08:00
Aliaksandr Valialkin	00a6ec85c6	docs/CHANGELOG.md: cut v1.87.0	2023-02-01 13:03:27 -08:00
Aliaksandr Valialkin	2d1ea90bcb	vendor: `make vendor-update`	2023-02-01 12:48:10 -08:00
Dmytro Kozlov	b7cc0bfe3c	app/vmctl: Add insecure skip verify flag for remote read protocol (#3611 ) * app/vmctl: Add insecure skip verify flag for remote read protocol	2023-02-01 12:45:11 -08:00
Aliaksandr Valialkin	ac695f36bb	lib/promscrape: add a comment explaining the logic behind adding `exported_` perfix to metric names This is a follow-up for `7b87fac8e7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3557 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2023-02-01 12:02:05 -08:00
Dmytro Kozlov	3c1e455805	lib/promscrape: fix `honor_labels` behavior (#3739 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-01 12:02:04 -08:00
Aliaksandr Valialkin	03145842d7	docs/CHANGELOG.md: document `9254e494f9`	2023-02-01 09:57:04 -08:00
Nikolay	554876cc38	lib/storage: fixes finalDedup for backfilled data (#3737 ) previously historical data backfilling may trigger force merge for previous month every hour it consumes cpu, disk io and decrease cluster performance. Following commit fixes it by applying deduplication for InMemoryParts	2023-02-01 09:57:02 -08:00
Zakhar Bessarab	326a702559	app/vmbackup: fix deleting snapshot after backup completion (#3735 ) (#3736 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-01 08:58:11 -08:00
Zakhar Bessarab	626bd22157	fix: vmselect multi-level setup panic (#3738 ) * app/vmselect/netstorage: fix panic for multi-level cluster setup when `replicationFactor` was set and request contained `trace` parameter (#3734) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmselect/netstorage: use correct context for retry Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-02-01 08:56:36 -08:00
Aliaksandr Valialkin	2049114e1f	app/vmselect/vmui: `make vmui-update` after `dcc5616126`	2023-01-31 13:24:54 -08:00
Aliaksandr Valialkin	e628289df4	docs/CHANGELOG.md: document `442a9f16b4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3661	2023-01-31 13:04:20 -08:00
Yury Molodov	89c4982658	vmui: improvement the theme (#3731 ) * feat: add detect the system theme * fix: change logic fetch tenants * feat: add docs and info to cardinality page --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-31 12:55:44 -08:00
Aliaksandr Valialkin	aae7b0dc97	vendor: `make vendor-update`	2023-01-31 11:05:11 -08:00
Aliaksandr Valialkin	a522bbc8b4	lib/bytesutil/internstring.go: increase the limit on the maximum string lengths, which can be interned The limit has been increased from 300 bytes to 500 bytes according to the collected production stats. This allows reducing CPU usage without significant increase of RAM usage in most practical cases. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692	2023-01-31 11:04:09 -08:00
Roman Khavronenko	d93ac2b1ea	docs: mention `-vmalert.proxyURL` in vmalert docs (#3730 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-31 10:49:49 -08:00
Aliaksandr Valialkin	855d560789	lib/promscrape/discovery/azure: add __meta_azure_machine_size label in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/11650	2023-01-27 17:07:57 -08:00
Aliaksandr Valialkin	134f7622d6	lib/promscrape/discovery/kubernetes: add support for __meta_kubernetes_pod_container_id See https://github.com/prometheus/prometheus/issues/11843 and https://github.com/prometheus/prometheus/pull/11844	2023-01-27 16:33:57 -08:00
Aliaksandr Valialkin	80fa0e2df4	vendor: `make vendor-update`	2023-01-27 15:59:29 -08:00
Yury Molodov	730025d1dc	vmui: add select of Tenant ID (#3673 ) * feat: add select of tenantID * feat: replace tenantID to default url * fix: move the tenantID selector to the top header * fix: hide tenantID selector by condition * fix: correct z-index * app/vmselect/vmui: `make vmui-update` --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-27 15:54:09 -08:00
Aliaksandr Valialkin	1df4f63bc8	docs: update security chapters after `bd716d1b0c`	2023-01-27 15:44:49 -08:00
Denys Holius	1259a931c8	Improving docs by adding additional security sections (#3713 ) * docs/Cluster-VictoriaMetrics.md: adds security section * docs/Quick-Start.md: adds Security recommendation section	2023-01-27 15:41:16 -08:00
Aliaksandr Valialkin	cd25945ff6	app/vmagent: properly return 200 response code when importing data via Prometheus PushGateway protocol This is the same fix as has been already applied to app/vminsert at `cdb6d651e9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2023-01-27 14:40:26 -08:00
Aliaksandr Valialkin	8838c07360	docs/vmauth.md: update docs after `ff39a91147` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346	2023-01-27 14:10:29 -08:00
Aliaksandr Valialkin	18bf18b8b9	app/vmauth: limit the number of concurrent requests served by `vmauth` with the `-maxConcurrentRequests` command-line flag See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346 This commit is based on the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486	2023-01-27 14:07:53 -08:00
Aliaksandr Valialkin	27ec56ffb1	app/vmauth: do not use net/http/httputil.ReverseProxy This allows better controlling requests to backends and providing better error logging. For example, if the backend was unavailable, then the ReverseProxy was logging the error message without client ip and the initial request uri. This could harden debugging. This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486	2023-01-27 13:40:53 -08:00
Aliaksandr Valialkin	bccbe07c33	lib/netutil: move IsTrivialNetworkError() function there, since it is used in multiple places across the code	2023-01-27 13:24:44 -08:00
Aliaksandr Valialkin	341026902e	app/vmauth: pass the target url to reverse proxy via context.Value instead of request header This is less hacky way, since it doesn't clash with request headers	2023-01-27 12:16:55 -08:00
Aliaksandr Valialkin	ae2b9f9204	docs/managed-victoriametrics: typo fix in links to images	2023-01-27 11:36:29 -08:00
Aliaksandr Valialkin	c13e341d16	docs/assets/README.md: mention that locally placed doc-specific images simplify referring them from various views without the need to deal with folder prefixes	2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin	8bde11e1a3	docs/managed-victoriametrics/user-management.md: move the associated images to docs/managed-victoriametrics/ folder with user-management_ prefix according to docs/assets/README.md	2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin	955e1c1729	docs/managed-victoriametrics/quickstart.md: move the associated images to docs/managed-victoriametrics/ folder with quickstart_ prefix according to docs/assets/README.md	2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin	b024981bed	docs/assets: add README.md with the explanation on which files can be put into the docs/assets folder	2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin	c06ec450ee	docs/Cluster-VictoriaMetrics.md: move Naive_cluster_scheme.png from the docs/assets/images/ folder into docs/ folder and add Cluster-VictoriaMetrics_ prefix to the image name The docs/assets folder should be used only for assets specific to docs generation at https://docs.victoriametrics.com, e.g. css, js and images. All the other assets related to specific docs should be placed in the same folder as the corresponding .md file. These assets should have the same name prefix as the corresponding doc file name. This simplifies tracking the lifetime of these assets. For example, if the doc is removed, it is very easy to remove all assets associated with it with a simple `rm -rf docs/doc-name` command. This also simplifies generating correct urls for doc-specific assets from both https://docs.victoriametrics.com and from https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/ - just refer to the asset name without any directory prefixes.	2023-01-27 11:31:21 -08:00
Aliaksandr Valialkin	eb10102521	lib/netutil: typo fix in the error message	2023-01-27 11:31:05 -08:00
dmitryk-dk	7cf9752a7f	docs: move how to register from dbaas to docs	2023-01-27 11:30:45 -08:00
dmitryk-dk	0d81aff91d	docs: use absolute path	2023-01-27 11:29:54 -08:00
dmitryk-dk	6ed4e70a00	docs: add documentation of user management on managed-vm	2023-01-27 11:26:36 -08:00
Aliaksandr Valialkin	7dc38fba72	app/vmauth: consistency renaming: UserInfo.URLMap -> UserInfo.URLMaps This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486	2023-01-27 00:19:46 -08:00
Aliaksandr Valialkin	9e8cfd23ad	docs/Cluster-VictoriaMetrics.md: update command-line descriptions after `ebebaecd94`	2023-01-27 00:04:54 -08:00
Aliaksandr Valialkin	4cf4c307ea	docs: update command-line descriptions after `73256fe438`	2023-01-27 00:01:14 -08:00
Aliaksandr Valialkin	b17857c7a4	lib/netutil: limit the time needed for reading proxy protocol headers This should prevent from misconfigured proxies and from possible Slowloris-type DoS attacks (see https://en.wikipedia.org/wiki/Slowloris_(computer_security) ) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335	2023-01-26 23:47:06 -08:00
Aliaksandr Valialkin	1a1f3e7149	app/vmagent/prometheusimport: delete the temporary directory created by vmagent after the test is complete This is a follow-up for `1cfa183c2b`	2023-01-26 23:25:34 -08:00
Nikolay	ebebaecd94	lib/netutil: init implimentation of proxy protocol (#3687 ) * lib/netutil: init implimentation of proxy protocol https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335 * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-26 23:25:22 -08:00
Aliaksandr Valialkin	e8ee219fba	docs/CHANGELOG.md: make the description for the bugfix from `465a285324` more reader-friendly	2023-01-26 10:08:19 -08:00
Nikolay	4af05065d1	lib/storage: properly release parts inMerge lock (#3711 ) if storage doesn't have enough disk space, finalDedupWatcher holds inMerge lock for all parts and never release it until storage restart	2023-01-26 08:57:36 -08:00
Roman Khavronenko	29ea196bf0	docs: specify the time window for `series_limit` (#3708 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-25 09:31:09 -08:00
Aliaksandr Valialkin	bd809db4d9	docs: update the list of command-line flags according to the latest changes	2023-01-25 09:22:23 -08:00
Aliaksandr Valialkin	5defa99a2e	lib/streamaggr: add ability to de-duplicate input samples before aggregation	2023-01-25 09:22:03 -08:00
Yury Molodov	29fd95d426	vmui: include fonts in its bundle (#3705 ) * feat: include fonts in the build * fix: reduce size fonts * wip - Document the change at docs/CHANGELOG.md - Run `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-24 09:31:37 -08:00
Yury Molodov	97aa9b03de	vmui: improvements to the UI styles (#3704 ) * feat: add dark theme * update packages * feat: add multilevel menu (#3678) * fix: correct styles * fix: update link to cardinality-explorer * fix: remove unused scss variables * docs/CHANGELOG.md: document the changes Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-24 09:21:20 -08:00
Roman Khavronenko	c24c2d20d4	discover/ec2: follow-up after `e2b4ab8384` (#3703 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-24 09:10:46 -08:00
Roman Khavronenko	dad25672e2	discover/ec2: bump API version (#3702 ) Switch to the actual API version `2016-11-15`, since the old version doesn't provide access to all the fields which implementation expects. For example, old API missing `zone_id` field in `DescribeAvailabilityZonesResponse` response. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-24 09:10:45 -08:00
Aliaksandr Valialkin	0698467ae5	lib/bytesutil: do not intern long strings, since they may need big amounts of additional memory for the cache Allow users fine-tuning the maximum string length for interning via -internStringMaxLen command-line flag. This may be used for fine-tuning RAM vs CPU usage for certain workloads. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692	2023-01-23 23:37:08 -08:00
Aliaksandr Valialkin	2ce69468cd	deployment/docker: sync with master branch	2023-01-23 22:59:42 -08:00
Roman Khavronenko	e75d7fefb4	dashboards: add non-default flags panel for vmagent (#3453 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-23 22:55:06 -08:00
Aliaksandr Valialkin	417b90b8ad	app/vmagent/{promremotewrite,vmimport}: remove unused functions InsertHandlerForReader() Thanks to `1cfa183c2b` , where the first such function has been removed	2023-01-23 22:42:46 -08:00
Aliaksandr Valialkin	4b3a207705	app/{vmagent,vminsert}: follow-up for `1cfa183c2b` - Call httpserver.GetQuotedRemoteAddr() and httpserver.GetRequestURI() only when the error occurs. This saves CPU time on fast path when there are no parsing errors. - Create a helper function - httpserver.LogError() - for logging the error with the request uri and remote addr context.	2023-01-23 22:41:08 -08:00
Artem Navoiev	0ac0cfdc69	add error handler for parsing prometheus text format to vmagent and v… (#3693 ) * add error handler for parsing prometheus text format to vmagent and vminsert Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix variables naming and error message Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-23 22:36:23 -08:00
Yury Molodov	bad7213fcb	vmui: add open graph and twitter card tags (#3697 ) * feat: add open graph and twitter card tags * app/vmui: spelling fixes Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-23 22:33:42 -08:00
Aliaksandr Valialkin	71a170d404	lib/promscrape: follow-up for `393876e52a` - Document the change in docs/CHANGELOG.md - Reduce memory usage when sending stale markers even more by parsing the response in stream parsing mode - Update the TestSendStaleSeries Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675	2023-01-23 21:56:18 -08:00
Roman Khavronenko	8e2a8a6ae2	lib/promscrape: limit number of sent stale series at once (#3686 ) Stale series are sent when there is a difference between current and previous scrapes. Those series which disappeared in the current scrape are marked as stale and sent to the remote storage. Sending stale series requires memory allocation and in case when too many series disappear in the same it could result in noticeable memory spike. For example, re-deploy of a big fleet of service can result into excessive memory usage for vmagent, because all the series with old pod name will be marked as stale and sent to the remote write storage. This change limits the number of stale series which can be sent at once, so memory usage remains steady. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-23 21:56:17 -08:00
Aliaksandr Valialkin	95d4db0506	lib/promscrape: properly log the actual response size after `c4229a1bba`	2023-01-23 21:13:06 -08:00
Aliaksandr Valialkin	903b2e710c	lib/storage: use deterministic random generator in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 20:12:32 -08:00
Aliaksandr Valialkin	4c7062b408	lib/mergeset: use deterministic random generator in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 19:44:10 -08:00
Aliaksandr Valialkin	f8dcbe4abd	lib/mergeset: fix data race in BenchmarkInmemoryBlockMarshal	2023-01-23 19:44:07 -08:00
Aliaksandr Valialkin	6ff15ca135	app/vmselect: use consistent randomizer in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 19:27:40 -08:00
Aliaksandr Valialkin	ef7683f2e0	app/vmalert: use consistent randomizer in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 19:25:32 -08:00
Aliaksandr Valialkin	107a056ade	lib/decimal: use consistent randomizer in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 19:24:05 -08:00
Aliaksandr Valialkin	796c7b0ee1	lib/uint64set: use repeatable randomizer in tests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 19:24:05 -08:00
Aliaksandr Valialkin	dfb1d1ead1	lib/encoding: make deterministic tests which rely on math/rand Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683	2023-01-23 18:43:49 -08:00
Aliaksandr Valialkin	bb1b87e5af	vendor: `make vendor-update`	2023-01-23 18:43:43 -08:00
Denys Holius	0f25a77acd	Fix/remove vmanomaly from release guide (#3699 ) * docs/Release-Guide.md: remove vmanomaly from release guide because it has own release cycle * fixed a typo	2023-01-23 08:08:02 -08:00
Aliaksandr Valialkin	f6d4fde45b	docs/Articles.md: added missing third-party articles about VictoriaMetrics	2023-01-22 14:23:55 -08:00
Aliaksandr Valialkin	7d3d664eb8	docs/Single-server-VictoriaMetrics.md: make it clear that VictoriaMetrics supports both pull and push protocols at `how to import time series data` chapter	2023-01-22 14:05:15 -08:00
Aliaksandr Valialkin	7f35d81cf4	docs/Articles.md: add https://dev.to/aws-builders/ultra-monitoring-with-victoria-metrics-1p2	2023-01-22 13:52:13 -08:00
Aliaksandr Valialkin	d8329e47cf	lib/vmselectapi: propagate timeout errors from vmselect to vmstorage instead of closing the connection established from vmselect to vmstorage This is a follow-up for `20e9598254`	2023-01-20 19:30:22 -08:00
Aliaksandr Valialkin	b046af8a4d	app/vmselect: `make vmui-update` after `df7b81b44d`	2023-01-20 12:07:29 -08:00
Yury Molodov	57f7ffca5b	vmui: add support for time zone selection for older versions of browsers (#3680 ) * fix: add check for support of getting time zones * vmui: add support for time zone selection for older versions of browsers	2023-01-20 12:07:28 -08:00
Denys Holius	4429e7e53e	Adds some improvements to release guide docs (#3679 ) * docs/Release-Guide.md: fixed a typo * Release-Guide.md: adds missed steps for updating vmanomaly and vmgateway helm charts	2023-01-19 10:01:54 -08:00
Max Golionko	28738c473a	ci: checkout correct branch for build step (#3676 )	2023-01-19 09:51:53 -08:00
Aliaksandr Valialkin	0d91e0f035	deployment/docker: update VictoriaMetrics components in docker-compose from v1.86.0 to v1.86.2	2023-01-18 12:57:52 -08:00
Aliaksandr Valialkin	069042653c	docs/CHANGELOG.md: cut v1.86.2	2023-01-18 12:01:14 -08:00
Aliaksandr Valialkin	62e08dbea9	.github/workflows: remove obsolete make targets: install-goling and install-errcheck These targets became obsolete after `ec2c82e800`	2023-01-18 11:48:47 -08:00
Max Golionko	57a0cde27c	CI: split js and go codeql, split test and build, enable matrix for test (#3670 ) * split js and go codeql, split test and build, enable matrix for test * checkout before go setup * enable build for PRs as well * update filter	2023-01-18 11:46:38 -08:00
Aliaksandr Valialkin	cd55e82208	docs/CHANGELOG.md: document `777038fe44` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3672	2023-01-18 11:40:53 -08:00
Tobias Jungel	8bdc63aab9	app/vmbackup: prevent password leaks (#3672 ) This prevents vmbackup from leaking passwords into logs like shown below. 2023-01-11T15:00:01.050Z info VictoriaMetrics/lib/logger/flag.go:12 build version: vmbackup-20221214-211706-tags-v1.85.1-0-g09a70d3e9 2023-01-11T15:00:01.050Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags 2023-01-11T15:00:01.050Z info VictoriaMetrics/lib/logger/flag.go:20 -dst="fs:///vm-backups/latest" 2023-01-11T15:00:01.050Z info VictoriaMetrics/lib/logger/flag.go:20 -snapshot.createURL="http://user:super_sercret123@victoriametricspshot/create" 2023-01-11T15:00:01.050Z info VictoriaMetrics/lib/logger/flag.go:20 -storageDataPath="/storage" 2023-01-11T15:00:01.050Z info VictoriaMetrics/app/vmbackup/main.go:53 Snapshot create url http://user:super_sercret123@victoriametrics:8428/snapshot/create 2023-01-11T15:00:01.050Z info VictoriaMetrics/app/vmbackup/main.go:60 Snapshot delete url http://user:super_sercret123@victoriametrics:8428/snapshot/delete	2023-01-18 11:40:52 -08:00
Aliaksandr Valialkin	46645f5d94	app/vmui: increase perceived performance by 2.5x by reducing the delay before the query execution from 0.8s to 0.3s The delay cannot be removed, since it is used for limiting the rate of queries sent to VictoriaMetrics during graph scrolling.	2023-01-18 01:33:51 -08:00
Aliaksandr Valialkin	c5e858461c	lib/{storage,mergeset}: wake up background merges as soon as there is a potential work for them Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647	2023-01-18 01:10:43 -08:00
Aliaksandr Valialkin	70b5a6fb28	lib/{storage,mergeset}: do not run assisted merges when flushing pending samples to parts Assisted merges are intended to be performed by goroutines, which accept the incoming samples, in order to limit the data ingestion rate. The worker, which converts pending samples to parts, shouldn't be penalized by assisted merges, since this may result in increased number of pending rows as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647#issuecomment-1385039142 when the assisted merge takes too much time.	2023-01-18 00:25:33 -08:00
Aliaksandr Valialkin	0c90b49e4b	lib/storage: use better naming for a function returning new []rawRows - newRawRowsBlock() -> newRawRows()	2023-01-18 00:01:21 -08:00
Aliaksandr Valialkin	e01f52d517	app/vmselect/promql: updates tests for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664	2023-01-17 23:26:06 -08:00
Aliaksandr Valialkin	a844b97942	lib/promscrape: follow-up for `d79f1b106c` - Document the fix at docs/CHANGELOG.md - Limit the concurrency for sendStaleMarkers() function in order to limit its memory usage when big number of targets disappear and staleness markers are sent for all the metrics exposed by these targets. - Make sure that the writeRequestCtx is returned to the pool when there is no need to send staleness markers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668	2023-01-17 23:13:08 -08:00
lzfhust	5ac0f18ca8	using writeRequestCtxPool when delete kubernetes clusters from kubernetes_sd_configs (#3669 )	2023-01-17 23:12:59 -08:00
Zakhar Bessarab	40d524edb8	discovery/{consul,nomad}: fix cancelling serviceWatcher in-flight requests (#3658 ) * lib/promscrape/discovery/{consul,nomad}: fix background service update watches not canceling requests on serviceWatcher stop Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/{consul,nomad}: fix closing serviseWatcher during scrape job restart Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * wip Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-17 21:47:51 -08:00
Scott Kevill	63653b53d6	lib/fs: use `unix.Statfs()` / `unix.Statvfs()` when using a path (#3663 )	2023-01-17 21:22:02 -08:00
Roman Khavronenko	09d41e05fa	ci: disable JS codeQL check (#3659 ) We have limited amount of time used by Github CI runners and JS analysis accounts for a half of it. Since JS represents only a small fraction of the codebase and is solely maintained by one person - I suggest to disable the CodeQL check in order to save CI runners time. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-17 21:05:51 -08:00
Yury Molodov	0561ba3557	vmui: correctly display range results in Table view (#3657 ) * fix: properly display range results * fix: set range values to empty array * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-17 21:04:50 -08:00
Aliaksandr Valialkin	d241a71baa	docs/CHANGELOG.md: fix the link to feature request implemented at `e58921aa8f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3571	2023-01-17 20:28:04 -08:00
Yury Molodov	6e033b93b0	vmui: give more visually different colors to graph lines (#3656 ) * feat: make more different colors of graph lines * docs/CHANGELOG.md: give more visually different colors to graph lines	2023-01-17 20:26:23 -08:00
Aliaksandr Valialkin	0498377837	app/vmui: do not round the number in formatPrettyNumber() if the range isn't set	2023-01-17 19:51:43 -08:00
Aliaksandr Valialkin	42fa6cee19	vendor: update github.com/VictoriaMetrics/metricsql from v0.51.1 to v0.51.2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664	2023-01-17 11:26:11 -08:00
Aliaksandr Valialkin	c33728befb	lib/promscrape: properly apply series limit Fix the following issues: - Series limit wasn't applied when staleness tracking was disabled. - Series limit didn't prevent from sending staleness markers for new series exceeding the limit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660 Thanks to @hagen1778 for the initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3665	2023-01-17 10:30:16 -08:00
Aliaksandr Valialkin	e370daecda	docs/CHANGELOG.md: document the 74f196c37813a18c2c8f28831696ed7d9f535604	2023-01-17 09:10:51 -08:00
Aliaksandr Valialkin	103dfd0525	lib/{mergeset,storage}: do not slow down concurrently executed queries during assisted merges Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2023-01-16 14:45:40 -08:00
Aliaksandr Valialkin	4fc8cacd1a	vendor: `make vendor-update`	2023-01-15 14:16:45 -08:00
Aliaksandr Valialkin	4294c9b720	github.com/VictoriaMetrics/metrics: update from v1.23.0 to v1.23.1 See https://github.com/VictoriaMetrics/metrics/issues/42	2023-01-15 14:07:14 -08:00
Yury Molodov	060780af69	vmui: make the step input field global across all the tabs and views (#3644 ) * feat: make the step input field global * fix: correct get step from url * fix: set minimumSignificantDigits to 1 * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-15 13:47:48 -08:00
Aliaksandr Valialkin	556484a52f	app/vminsert: return 200 OK status code when importing data in pushgateway format Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636	2023-01-15 13:26:09 -08:00
Denys Holius	41ea90e6a2	guides/guide-delete-or-replace-metrics.md: fixed wrong curl command (#3652 )	2023-01-15 13:03:32 -08:00
Yury Molodov	e37bac07a0	feat: make nav menu as links (#3646 )	2023-01-15 13:01:56 -08:00
Corporte Gadfly	5815a98957	docs: fix typo (#3648 )	2023-01-15 13:01:23 -08:00
Aliaksandr Valialkin	ce47faf102	app/vmselect/promql: reduce memory allocations when searching for time series pairs with identical labelsets in `q1 op q2` queries Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641	2023-01-15 13:00:28 -08:00
Aliaksandr Valialkin	fe8802bbc8	app/vmselect/promql: reduce the number of memory allocations inside getCommonLabelFilters() This should improve performance a bit for `q1 op q2` queries	2023-01-15 12:56:21 -08:00
Aliaksandr Valialkin	26f6cfd3b2	app/vmselect/netstorage: tune the number of blocks per series which should be unpacked by a single goroutine instead of spinning up multiple goroutines This reduces overhead on time series data unpacking for typical cases, this reducing CPU usage at vmselect Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641	2023-01-12 09:35:15 -08:00
Aliaksandr Valialkin	b735d49625	docs/CHANGELOG.md: update the description of the change at `20f28eb9d6`	2023-01-12 09:00:07 -08:00
Nikolay	43d1f2d0c4	/lib/promscrape: use correct err logger for scrape unmarshalling (#3645 ) /lib/promscrape: use correct err logger for scrape unmarshalling It correctly suppresses scrape errors and adds correct context for err msg	2023-01-12 09:00:06 -08:00
Roman Khavronenko	0a2e4d1939	dashboards: bump operator dash to v9 of Grafana (#3642 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-12 08:49:09 -08:00
Aliaksandr Valialkin	41b0b951f3	app/vmselect/netstorage: unpack series blocks in the current goroutine if their count doesnt exceed 100 This should improve performance a bit for common case	2023-01-12 01:31:38 -08:00
Aliaksandr Valialkin	d33a65e401	app/vmselect/promql: reduce memory allocations at getCommonLabelFilters() function Intern tag keys and values there	2023-01-12 01:27:34 -08:00
Aliaksandr Valialkin	a819e30ddf	lib/promscrape: log the number of unsuccessful scrapes during the last -promscrape.suppressScrapeErrorsDelay This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3413 Thanks to @jelmd for the pull request. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575	2023-01-12 01:12:22 -08:00
Aliaksandr Valialkin	a6988eb8c3	docs/Cluster-VictoriaMetrics.md: mention the -vmui.customDashboardsPath command-line flag at vmselect Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322	2023-01-11 23:39:18 -08:00
Aliaksandr Valialkin	820357f434	app/vmselect: follow-up after `820312a2b1` - Move the feature description at the correct place at docs/CHANGELOG.md - Run `make vmui-update` - Various cosmetic fixes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322	2023-01-11 23:37:51 -08:00
Dmytro Kozlov	f2862f088b	app/vmui: define custom path for dashboards json file (#3545 ) * app/vmui: define custom path for dashboards json file * app/vmui: remove unneeded code * app/vmui: move handler to own file, fix show dashboards, * app/vmui: move flag to handler, add flag description * app/vmauth: fix part of the comments * feat: add store for dashboards * fix: prevent fetch dashboards for app mode * app/vmauth: use simple cache for predefined dashboards * app/vmauth: update dashboards doc * app/vmauth: fix ci * app/vmui: decrease timeout * app/vmselect: removed cache, fix comments * app/vmselect: remove unused const * app/vmselect: fix error log, use slice byte instead of struct Co-authored-by: Yury Moladau <yurymolodov@gmail.com>	2023-01-11 23:33:30 -08:00
Dmytro Kozlov	99fbb2948b	app/vmctl: add remote read protocol integration tests (#3626 )	2023-01-11 23:03:17 -08:00
Aliaksandr Valialkin	2e018aebf3	lib/promscrape/discovery: missing changes after `b4ad3a3b4c`	2023-01-11 23:03:14 -08:00
Aliaksandr Valialkin	434f22f871	lib/promscrape: follow-up for `8537533beb` - Add a comment describing the purpose of the `role` field inside `apiConfig` struct - Revert changes at lib/promscrape/discovery/dockerswarm/dockerswarm.go , since they reduce code readability. E.g. the reader needs to look up the named string constants in order to get their values.	2023-01-11 22:56:48 -08:00
Zakhar Bessarab	ae5b85966a	lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects (#3632 ) * lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/CHANGELOG.md Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-11 22:56:40 -08:00
Yury Molodov	4295ca2ce2	vmui: small changes on explore metrics page (#3634 ) * fix: change issue link * fix: remove legend toggle * fix: move select graph size * feat: save url params on explore metrics page * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-11 22:17:06 -08:00
Artem Navoiev	efec0f150f	fix operator docs hugo schema Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-11 17:00:14 -08:00
Artem Navoiev	8d238f1bc2	docs: prepare operator docs to migration Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-11 16:59:52 -08:00
Artem Navoiev	3991c977af	docs: operator .MD -> .md Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-11 16:59:24 -08:00
Roman Khavronenko	17b5ac7e1c	dasbhoards: fix the tooltip info for 1.86 (#3628 ) See `c63755c316 (diff-bba263a473e7fbc9d0fde075ebef6b3d4e32c322ee1210a3e07182292c7723aaR18)` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-11 16:59:02 -08:00
Aliaksandr Valialkin	23ed835043	docs/CHANGELOG.md: cut v1.86.1	2023-01-11 01:27:01 -08:00
Artem Navoiev	e91814e332	vmagent: add minimal scrape file exampe for vmagent quick start (#3627 ) * vmagent: add minimal scrape file exampe for vmagent quick start Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * replace example with link to your prometheus.yml in docker Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-11 01:23:07 -08:00
Aliaksandr Valialkin	af58ac25f6	lib/vmselectapi: properly calculate query timeout vmselect passes query timeout to vmstorage in seconds. The commit `20e9598254` treated it as timeout in nanoseconds. Fix this in order to prevent from the following errors under vmstorage load: cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed.	2023-01-11 01:21:55 -08:00
Aliaksandr Valialkin	f7130d571d	app/vmselect: improve logging when the incoming query cannot be executed because of timeout in the wait queue	2023-01-11 01:12:25 -08:00
Aliaksandr Valialkin	675e0fa0ee	app/vmselect/promql: typo fix after `0771d57860`	2023-01-11 01:06:18 -08:00
Aliaksandr Valialkin	3d22532bb8	app/vmselect/promql: make a copy of per-series timestamps before their modification The per-series timestamps are usually shared among series, so it is unsafe modifying them. The issue has been appeared after the optimization at `2f3ddd4884`	2023-01-11 00:59:02 -08:00
Aliaksandr Valialkin	8a35377cf3	app/vmselect/promql: move the `eval function args in parallel` query trace outside the loop	2023-01-10 22:23:43 -08:00
Aliaksandr Valialkin	d1a62af3bd	docs/CHANGELOG.md: document v1.79.7 LTS release	2023-01-10 21:24:44 -08:00
Aliaksandr Valialkin	c6dbc4eb23	vendor: `make vendor-update`	2023-01-10 18:58:50 -08:00
Aliaksandr Valialkin	6d1bd761f8	deployment/docker: update Go builder from v1.19.4 to v1.19.5 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.5+label%3ACherryPickApproved	2023-01-10 18:42:34 -08:00
Aliaksandr Valialkin	e51d28aae2	docs/CHANGELOG.md: add release date for v1.86.0	2023-01-10 17:46:13 -08:00
Aliaksandr Valialkin	51b4f3c49f	deployment/docker: update Docker tag for VictoriaMetrics components from v1.85.3 to v1.86.0	2023-01-10 17:46:12 -08:00
Aliaksandr Valialkin	a2488f9e24	docs/CHANGELOG.md: cut v1.86.0	2023-01-10 16:22:53 -08:00
Aliaksandr Valialkin	ee73c9a5e6	deployment/docker: update Alpine base image from v3.17.0 to v3.17.1 See https://alpinelinux.org/posts/Alpine-3.17.1-released.html	2023-01-10 16:15:06 -08:00
Aliaksandr Valialkin	aa027529eb	lib/httpserver: directly pass flag value to CheckAuthFlag() There is no sense in passing a pointer to flag value there. This is a follow-up for `4225a0bd75`	2023-01-10 15:59:55 -08:00
Zakhar Bessarab	10f314cdbd	Use `httpAuth.` flags as a fallback for endpoints protected by `AuthKey` flags (#3582 ) * {lib/server, app/}: use `httpAuth.` flag as fallback for `AuthKey` if it is not set * lib/ingestserver/opentsdbhttp: fix opentdb HTTP handler not respecting `httpAuth.` flags Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-10 15:57:55 -08:00
Aliaksandr Valialkin	ab318660cd	lib/promscrape/discovery/gce: follow-up for `b2ccdaaa2f` - Use promutils.Labels.GetLabels() instead of comparing promutils.Labels.Labels to nil. This make the code more consistent with other places. - Mention the release where the issue has been introduced at docs/CHANGELOG.md. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624	2023-01-10 13:51:57 -08:00
Zakhar Bessarab	02f5c16433	lib/promscrape/discovery/gce: fix crash in case instance does not have any labels set (#3625 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-10 13:51:35 -08:00
Denys Holius	ceea223caa	docs/Release-Guide.md: added missed link to rpm repository (#3623 )	2023-01-10 13:43:45 -08:00
Aliaksandr Valialkin	98931449c1	app/vmselect/netstorage: reduce tail latency during query processing Previously the selected time series were split evenly among available CPU cores for further processing - e.g unpacking the data and applying the given rollup function to the unpacked data. Some time series could be processed slower than others. This could result in uneven work distribution among available CPU cores, e.g. some CPU cores could complete their work sooner than others. This could slow down query execution. The new algorithm allows stealing time series to process from other CPU cores when all the local work is done. This should reduce the maximum time needed for query execution (aka tail latency). The new algorithm should also scale better on systems with many CPU cores, since every CPU processes locally assigned time series without inter-CPU communications. The inter-CPU communications are used only when all the local work is finished and the pending work from other CPUs needs to be stealed.	2023-01-10 13:42:26 -08:00
Aliaksandr Valialkin	158a280822	app/vmselect/netstorage: reduce memory allocations when unpacking time series Unpack time series with less than 4M samples in the currently running goroutine. Previously a new goroutine was being started for unpacking the samples. This was requiring additional memory allocations.	2023-01-09 23:17:34 -08:00
Aliaksandr Valialkin	c8bd3534cb	app/vmselect/promql: eliminate memory allocation when sorting values inside float64s	2023-01-09 23:06:57 -08:00
Aliaksandr Valialkin	7956b0d974	app/vmselect/promql: pre-allocate memory for values to be merged in mergeTimeseries() This should reduce the number of memory re-allocations	2023-01-09 22:52:19 -08:00
Aliaksandr Valialkin	895d5d9d22	app/vmselect/promql: consistently intern series names obtained from marshalMetricNameSorted This reduces memory allocations when the returned series names are used as map keys later	2023-01-09 22:46:30 -08:00
Aliaksandr Valialkin	12e2bcdf81	app/vmselect/promql: avoid memory allocations and copying from source timeseries to the returned result at timeseriesToResult()	2023-01-09 22:39:15 -08:00
Aliaksandr Valialkin	dd92e2050f	app/vmselect/promql: remove memory allocations from sortMetricTags()	2023-01-09 22:22:58 -08:00
Aliaksandr Valialkin	8050f5a18c	app/vmselect/promql: intern output series names inside timeseriesToResult() This reduces the number of memory allocations for repeated queries, which return (almost) the same set of time series.	2023-01-09 22:20:34 -08:00
Aliaksandr Valialkin	efb3c630fe	app/vmselect/promql: intern output series names during normal aggregation	2023-01-09 22:15:31 -08:00
Aliaksandr Valialkin	c0de651558	app/vmselect/promql: intern output series names during incremental aggregation This should reduce the number of memory allocations for repeated queries	2023-01-09 22:12:05 -08:00
Aliaksandr Valialkin	abbac2c27c	app/vmselect/netstorage: pre-allocate 4 block references per each time series during querying Usually the number of blocks returned per each time series during queries is around 4. So it is a good idea to pre-allocate 4 block references per time series in order to reduce the number of memory allocations.	2023-01-09 22:08:30 -08:00
Aliaksandr Valialkin	2483c67579	app/vmselect/netstorage: cache canonical MetricName for time series returned from the storage This reduces memory allocations for repeated queries, which return (almost) the same set of time series.	2023-01-09 21:56:27 -08:00
Aliaksandr Valialkin	b7a4650ab0	all: use metricsql.CompileRegexp instead of regexp.Compile for compiling regexps used in graphite queries This should speed up repeated queries, since metricsql.CompileRegexp returns regexps from the cache on subsequent calls for the same input regexp.	2023-01-09 21:45:34 -08:00
Aliaksandr Valialkin	3d2cae9d75	vendor: `make vendor-update`	2023-01-09 21:34:44 -08:00
Aliaksandr Valialkin	43a4dcdaf8	lib/promscrape/discovery/nomad: sync nomad_sd_configs fields with the Prometheus implementation See the list of configs supported by Prometheus at `f88a0a7d83/discovery/nomad/nomad.go (L76-L84)` - Removed "token" option. In can be set either via NOMAD_TOKEN env var or via `bearer_token` config option. - Removed "scheme" option. It is automatically detected depending on whether the `tls_config` is set. - Removed "services" and "tags" options, since they aren't supported by Prometheus. - Added "region" option. If it is missing, then the region is read from NOMAD_REGION env var. If this var is empty, then it is set to "global" in the same way as Nomad client does. See `865ee8d37c/api/api.go (L297)` and `865ee8d37c/api/api.go (L555-L556)` - If the "server" option is missing, then it is read from NOMAD_ADDR in the same way as Nomad client does - see `865ee8d37c/api/api.go (L294-L296)` This is a follow-up for `8aee209c53` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-09 21:30:19 -08:00
Aliaksandr Valialkin	9f02f5a05a	app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks()	2023-01-09 21:29:01 -08:00
Denys Holius	e58fecb019	deployment/docker: update Alertmanager tag from v0.24.0 to v0.25.0 in docker-compose files (#3619 ) deployment/docker: bump alertmanager to latest v0.25.0	2023-01-09 21:25:10 -08:00
Roman Khavronenko	ca5136a0ee	lib/promscrape: remove `datacenter` field from nomad_sd_config (#3612 ) Looks like `datacenter` field isn't part of `/v1/services` API. See https://developer.hashicorp.com/nomad/api-docs/services#list-services and https://developer.hashicorp.com/nomad/api-docs/services#read-service Related issues: https://github.com/traefik/traefik/issues/9109 https://github.com/prometheus/prometheus/issues/11776 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-09 21:24:46 -08:00
Aliaksandr Valialkin	96f04c9863	app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333 This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 , but doesn't slow down the common case with merging replicated data blocks so significantly. Benchmark results: Before the change: BenchmarkMergeSortBlocks/replicationFactor-1-4 13968 85643 ns/op 956.53 MB/s 1700 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-2-4 10806 109171 ns/op 1500.77 MB/s 2191 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-3-4 8887 130623 ns/op 1881.45 MB/s 2660 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-4-4 7440 157348 ns/op 2082.52 MB/s 3174 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-5-4 6534 184473 ns/op 2220.38 MB/s 3612 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4 13419 85205 ns/op 961.44 MB/s 2213 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 579 1894900 ns/op 43.23 MB/s 46760 B/op 1 allocs/op After the change: BenchmarkMergeSortBlocks/replicationFactor-1-4 13832 85298 ns/op 960.40 MB/s 1716 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-2-4 8833 134222 ns/op 1220.66 MB/s 2675 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-3-4 6487 184830 ns/op 1329.65 MB/s 3636 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-4-4 4977 236318 ns/op 1386.61 MB/s 4733 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-5-4 4088 296734 ns/op 1380.36 MB/s 5761 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4 14083 84067 ns/op 974.47 MB/s 2110 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 536 2043534 ns/op 40.09 MB/s 50511 B/op 1 allocs/op	2023-01-09 12:58:18 -08:00
Aliaksandr Valialkin	7792ba3272	lib/promscrape/discoveryutils: cleanup after `5df9fddaf2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468	2023-01-07 01:27:16 -08:00
Zakhar Bessarab	e8624fd781	lib/promscrape/discoveryutils: use correct timeout for blocking requests (#3609 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-07 01:27:10 -08:00
Aliaksandr Valialkin	eb9a542c1f	lib/storage: simplify the fix from `488940502c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3566	2023-01-07 01:11:35 -08:00
Dmytro Kozlov	f739e44802	lib/storage: fix returning camelcase label names (#3608 ) * lib/storage: fix returning camelcase label names * doc: add change log * Update docs/CHANGELOG.md * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-07 01:11:10 -08:00
Aliaksandr Valialkin	c630115be0	lib/streamaggr: limit the the number of concurrent flushes of the aggregate data to the exact number of available CPUs This should reduce the maximum memory usage during concurrent flushes of the aggregate data	2023-01-07 00:19:34 -08:00
Aliaksandr Valialkin	0a14b7bb82	lib/promscrape: reduce the number of concurrently executed processScrapedData calls from 2x of the number of CPUs to the number of CPUs This should reduce the maximum memory usage for processScrapedData() function by 2x. The only part, which can be IO-bound in the processScrapedData() is pushData() call, when it buffers data to persistent queue if the remote storage cannot keep up with the data ingestion speed. In this case it is OK if the scrape pace will be limited.	2023-01-07 00:17:52 -08:00
Aliaksandr Valialkin	5876821a16	all: small improvements in error messages and command-line flag descriptions related to concurrency limiters	2023-01-07 00:12:24 -08:00
Aliaksandr Valialkin	3864357772	lib/writeconcurrencylimiter: moved the error generation from incConcurrency() to the caller place	2023-01-07 00:01:44 -08:00
Aliaksandr Valialkin	7fb02f536a	lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples This should reduce memory usage when scraping big number of targets, since this limits the summary memory usage during concurrent parsing and relabeling by the number of available CPU cores.	2023-01-06 23:01:18 -08:00
Aliaksandr Valialkin	3461ae8f13	lib/streamaggr: limit the number of concurrent flushes of aggregate metrics in order to limit memory usage	2023-01-06 22:40:19 -08:00
Aliaksandr Valialkin	2ca48444e2	lib/vmselectapi: typo fix after `20e9598254`	2023-01-06 22:13:32 -08:00
Aliaksandr Valialkin	b275983403	lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit Previously the -maxConcurrentInserts was limiting the number of established client connections, which write data to VictoriaMetrics. Some of these connections could be idle. Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting the number of such connections. So now the -maxConcurrentInserts command-line option limits the number of concurrently executed insert requests, not including idle connections. It is recommended removing -maxConcurrentInserts command-line option, since the default value for this option should work good for most cases.	2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin	20e9598254	lib/vmselectapi: limit the number of concurrently executed requests This should prevent from out of memory errors when big number of vmselect nodes send many concurrent requests to vmstorage The limit can be controlled at vmstorage via the following command-line flags: - search.maxConcurrentRequests - search.maxQueueDuration See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits	2023-01-06 18:39:46 -08:00
Aliaksandr Valialkin	cd705b0f69	app/vmselect: improve error message when the request cannot be started because too many concurrent requests are already executed	2023-01-06 18:19:05 -08:00
Aliaksandr Valialkin	be896ddfd4	lib/protoparser/clusternative: typo fix in the comment: thic -> this	2023-01-06 18:16:25 -08:00
Aliaksandr Valialkin	ec7a3b79ab	lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 21:53:08 -08:00
Aliaksandr Valialkin	54410bf51b	lib/promscrape: follow-up after `bced9fb978` - Document the bugfix at docs/CHANGELOG.md - Wait until all the worker goroutines are done in consulWatcher.mustStop() - Do not log `context canceled` errors when discovering consul serviceNames - Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client, since this handling is automatically performed by net/http.Transport. See DisableCompression option at https://pkg.go.dev/net/http#Transport . - Remove explicit handling of the proxyURL, since it is automatically handled by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport . - Expliticly set MaxIdleConnsPerHost, since its default value equals to 2. Such a small value may result in excess tcp connection churn when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client. - Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client. - Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468	2023-01-05 21:23:21 -08:00
Zakhar Bessarab	de5aad2cde	lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568 )	2023-01-05 21:23:15 -08:00
Roman Khavronenko	57277ed6bc	vmstorage: add more context to the flock acquiring msg (#3584 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3578 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-05 18:32:53 -08:00
Aliaksandr Valialkin	750d309f63	lib/promscrape/discovery/nomad: follow-up after `48f371a46c` - Remove undocumented `username` and `password` config options from `nomad_sd_config`. TODO: probably, remove these options from `consul_sd_config` too? These options exist there for backwards compatibility purposes. - Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels These labels contain AllocID and JobID fields for the discovered Nomad services. - Various typo fixes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 18:09:23 -08:00
Aliaksandr Valialkin	87a4943bea	Makefile: remove trailing space after `golangci-lint run` command It is left after `ec2c82e800`	2023-01-05 18:09:05 -08:00
Karan Sharma	8f42c5a024	lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549 ) Add nomad_sd_config support for service discovery	2023-01-05 18:07:02 -08:00
Luke Palmer	51d956225d	Lint and errcheck using golangci-lint (#3558 )	2023-01-05 17:00:17 -08:00
Zakhar Bessarab	eee7e75278	doc: add vmbackupmanager monitoring section (#3605 ) * doc: add vmbackupmanager monitoring section Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-05 16:49:59 -08:00
Thomas Danielsson	ec1f6811a1	dashboards: fix operator datasource variable (#3604 ) Got "Failed to upgrade legacy queries Datasource $ds was not found" in Grafana on operator dashboard. It's datasource variable was incorrectly named `datasource`. Also made the rest of the dashboards have homogeneous datasource-variable names and selections, matching vmagent dashboard.	2023-01-05 16:49:19 -08:00
Artem Navoiev	9d85c62ce2	Add Understand Your Setup Size Guide (#3572 ) docs: add Understand Your Setup Size Guide Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-05 16:48:28 -08:00
Aliaksandr Valialkin	6eda4c6da2	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2023-01-05 16:46:03 -08:00
Aliaksandr Valialkin	1af6e0b233	lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 This allows copy-n-pasting the url to another browser window and seeing the same result. The limit in 1000 chars is selected in order to prevent from potential issues with systems which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string If the limit is exceeded, then query args are sent via POST method and aren't visible in the url. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 16:45:42 -08:00
Aliaksandr Valialkin	21be927afa	docs/CHANGELOG.md: add missing dot	2023-01-05 03:35:20 -08:00
Zakhar Bessarab	99f9b02283	lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin	1d16cc9349	lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin	634e24e685	lib/promscrape: follow-up for `a7e29c38bc` - Document the bugfix at docs/CHANGELOG.md - Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 02:51:45 -08:00
Zakhar Bessarab	52226c392f	lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595 ) * lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/targetstatus: address review comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-05 02:48:39 -08:00
Yury Molodov	802febab74	vmui: improve `Explore metrics` (#3598 ) * feat: add multiple select * feat: improve explore interface * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-05 02:24:05 -08:00
Aliaksandr Valialkin	c97d6ed6a4	lib/streamaggr: sort `by` and `without` labels in the aggregate output metric name Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460	2023-01-05 02:08:59 -08:00
Aliaksandr Valialkin	78114e85d6	vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0 Updates https://github.com/VictoriaMetrics/metricsql/pull/7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589	2023-01-05 01:50:26 -08:00
Roman Khavronenko	7c79a6f4ac	dashboards: add backupmanager dashboard (#3599 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-04 13:34:04 -08:00
Aliaksandr Valialkin	ccec8c26ed	lib/streamaggr: remove unused fields	2023-01-04 13:33:21 -08:00
Roman Khavronenko	23473de81e	github: rm `plaintext` render (#3597 ) `render: plain text` makes fields not formattable and prevents from pasting screenshots. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-04 08:44:32 -08:00
Denys Holius	28649f0773	docs/Release-Guide.md: adds release scenario for RPM LTS packages (#3588 )	2023-01-04 08:39:10 -08:00
Zakhar Bessarab	e38100fbb5	github: use github templates for filling in feature requests or bug reports (#3587 ) github: use github templates for filling in feature requests or bug reports	2023-01-04 08:27:59 -08:00
Artem Navoiev	49f30320d6	update year in License Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-04 08:20:14 -08:00
Aliaksandr Valialkin	0c7d161858	vendor: `make vendor-update`	2023-01-03 23:36:36 -08:00
Aliaksandr Valialkin	ebb8aeb0cf	app/vmselect: remove dependency on lib/promscrape from app/vmselect	2023-01-03 23:27:36 -08:00
Aliaksandr Valialkin	ac890b3081	docs: update `-help` outputs for vm* tools	2023-01-03 23:27:31 -08:00
Aliaksandr Valialkin	3369371636	app/{vmagent,vminsert}: add support for streaming aggregation See https://docs.victoriametrics.com/stream-aggregation.html Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460	2023-01-03 22:22:07 -08:00
Aliaksandr Valialkin	d3f8298739	lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..))	2023-01-03 22:15:49 -08:00
Aliaksandr Valialkin	cfba42091e	app/vmagent/remotewrite: improve descriptions for `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` - Cross-reference these command-line flags. - Add a link to https://docs.victoriametrics.com/vmagent.html#relabeling	2023-01-03 22:05:39 -08:00
Aliaksandr Valialkin	b49927bcb1	docs/Single-server-VictoriaMetrics.md: impromve formatting for `prominent features` chapter	2023-01-03 21:58:38 -08:00
Aliaksandr Valialkin	7b700adf67	app/vmbackup: remove superflouos whitespace after `8fe21ec707`	2023-01-03 21:53:57 -08:00
Aliaksandr Valialkin	ff3e8d096d	docs/vmbackupmanager.md: run `make docs-sync` after `fa842d6534`	2023-01-03 21:53:57 -08:00
yanggang	dad6d1fbf6	Fix flag help message for the backups types. (#3577 ) Signed-off-by: yanggang <gang.yang@daocloud.io>	2023-01-03 21:52:49 -08:00
yanggang	f330cb11ba	fix typo for json values. (#3576 ) Signed-off-by: yanggang <gang.yang@daocloud.io>	2023-01-03 21:50:51 -08:00
yanggang	6e5bc3ed78	Fix vmctl command hint for vm-native-step-interval (#3575 ) Signed-off-by: yanggang <gang.yang@daocloud.io>	2023-01-03 21:49:15 -08:00
Artem Navoiev	557b9b1d57	run checks only for master/cluster branches (#3581 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-03 21:46:49 -08:00
Roman Khavronenko	dde750e7c1	vmalert: mention specifics of Alertmanager HA mode (#3573 ) Stress the importance of specifying of all Alertmanager URLs in vmalert's `-notifier.url` or `notifier.config` if it runs in cluster mode. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3547 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-03 21:46:00 -08:00
Aliaksandr Valialkin	a7942c6c0d	lib/promrelabel: allow calling Match on nil IfExpression This simplifies the caller side of IfExpression	2022-12-30 16:47:59 -08:00
Aliaksandr Valialkin	18ee36f0cf	app/vmui: reuse the series color in line tooltip	2022-12-29 15:33:09 -08:00
Aliaksandr Valialkin	d794e971fc	app/vmui: small usability improvements - Show in the line tooltip the number of the query which generates the given line. This simplifies comparison of lines generated by multiple queries. - Show metric name as __name__ label in the line tooltip in the same way as other labels are shown there. This makes the label information in the tooltip more consistent. - Properly quote label values with JSON.stringify(). This prevents from improper formatting when label values contain doublequote chars. - Remove double curly braces artifact at graph legend for lines without names and labels. - Properly use modifier for regular expressions across the code.	2022-12-29 15:00:10 -08:00
Aliaksandr Valialkin	c03adc2bce	docs/CHANGELOG.md: document `1720bddb4f` Updats https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3530	2022-12-29 12:32:05 -08:00
Aliaksandr Valialkin	71f2979669	app/vmselect/vmui: `make vmui-update` after `1720bddb4f`	2022-12-29 12:20:20 -08:00
Yury Molodov	2370f03c52	fix: display correct graph tooltip title (#3562 )	2022-12-29 12:20:18 -08:00
Roman Khavronenko	c22122212d	csvimport: support empty values (#3565 ) Before, if the imported line contained multiple metrics and one or more of them had an empty values - the whole line was ignored. Now, only metrics with empty values are ignored, and the rest of the metrics are accepted successfully. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-29 11:56:02 -08:00
Aliaksandr Valialkin	a6d4596d84	docs/CHANGELOG.md: add a link to the docs and the pull request for `update_entries_limit` option in vmalert This is a follow up for `6588fcbfca`	2022-12-29 10:41:51 -08:00
Aliaksandr Valialkin	5e87ebabc7	vendor: update github.com/valyala/fastjson/fastfloat from v1.6.3 to v1.6.4 This should properly parse floating-point numbers with missing integer or fractional parts. For example, 123. or .123 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544	2022-12-29 10:41:51 -08:00
ChenyuanHu	8dfe95761e	app/vmselect/prometheus: no need manually call queryDuration.UpdateDuration (#3564 ) There is no need to manually call `queryDuration.UpdateDuration(startTime)`, because `defer queryDuration.UpdateDuration(startTime)` is executed at the beginning of the function(L660).	2022-12-29 10:41:51 -08:00
Roman Khavronenko	5cf2998af8	vmalert: allow configuring the default number of stored rule's update states (#3556 ) Allow configuring the default number of stored rule's update states in memory via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's `update_entries_limit` configuration param. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-29 10:41:51 -08:00
Aliaksandr Valialkin	f27bb19213	app/vmselect/searchutils: accept partial RFC3339 values at `time`, `start` and `end` query args This simplifies manual usage of the APIs. For example, the following query would return the results over the 2022 year. /api/v1/query_range?start=2022&end=2023&step=1d&query=... This is equivalent to: /api/v1/query_range?start=2022-01-01T00:00:00Z&end=2023-01-01T00:00:00Z&step=1d&query=...	2022-12-28 19:46:21 -08:00
Yury Molodov	6f21435d2d	vmui: fix step field (#3561 ) * feat: use a unit next to the step value * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-28 16:01:33 -08:00
Aliaksandr Valialkin	7e49818c6d	vendor: `make vendor-update`	2022-12-28 15:00:25 -08:00
Aliaksandr Valialkin	8e9548f050	lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize This is a follow-up for `7ad9fff7e5`	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	8dc04a86f6	lib/{storage,mergeset}: tune the threshold for assisted merge The https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425#issuecomment-1359117221 reveals that CPU usage for incoming queries may significantly increase when the number of in-memory parts becomes too big. This commit reduces the maximum number of in-memory parts before starting the assisted merge during data ingestion. This should reduce CPU usage for incoming queries, since they need to inspect lower number of in-memory parts. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-28 14:42:45 -08:00
Clément Nussbaumer	04d536c15a	fix(promscrape): check MaxScrapeSize after gzip decompression (#3550 )	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	ae0a77d778	lib/snapshot: improve log message on unexpected status code during attempts to create or delete snapshots Use "unexpected status code returned from %q: %d; expecting %d" log message format instead of less clear format "unexpected status code returned from %q; expecting %d; got %d" This is a follow-up for `c612bb165e`	2022-12-28 11:46:23 -08:00
Aliaksandr Valialkin	824fd70ec2	docs: remove a case study for Dreamteam.gg, since it looks like the company no longer exists	2022-12-28 11:45:38 -08:00
Zakhar Bessarab	990c874b25	lib/snapshot: fix error message format for failed HTTP request (#3559 )	2022-12-28 11:44:06 -08:00
Artem Navoiev	1ae98867a5	fix broken links Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-12-28 11:30:53 -08:00
Artem Navoiev	393f4ab86f	update links to grafana dashboards (#3534 ) docs: update links to grafana dashboards Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-12-28 11:22:02 -08:00
Aliaksandr Valialkin	197e58f1f6	app/vmui: show min, max and avg lines at `Explore metrics` graphs when `instance` is selected in the same way as when only the `job` is selected This improves consistency of the graphs. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386	2022-12-23 23:21:23 -08:00
Aliaksandr Valialkin	eaaae2fab8	app/vmselect/vmui: `make vmui-update` after `0dca224ec3`	2022-12-23 22:26:23 -08:00
Aliaksandr Valialkin	182e62acde	app/vmui: small improvements for header panel - Rename `Custom panel` tab to more clear `Query` tab - Rename `Cardinality` tab to `Explore cardinality`, so it becomes consistent with `Explore metrics` tab - Move `Dashboards` tab to the end, since it isn't used too much	2022-12-23 22:26:22 -08:00
Aliaksandr Valialkin	8d03a5b1c0	app/vmui: move the `Explore metrics` tab closer to `Custom panel` tab Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386	2022-12-23 22:26:08 -08:00
Aliaksandr Valialkin	8579aecd77	app/vmui: show less lines at metrics explorer when the instance isn't selected Show min, max and avg graphs across instances for the selected job. This should improve usability of such a graphs when the job contains many instances. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386	2022-12-23 22:26:08 -08:00
Aliaksandr Valialkin	8a1775f3a6	app/vmui: follow-up after `f6d31f5216` - Document the feature at docs/CHANGELOG.md. - Document the metrics explorer at https://docs.victoriametrics.com/#metrics-explorer . - Properly set `start` and `end` args for the selected time range when performing the request, which returns metric names. - Improve queries, so they return lower number of lines and labels. This should improve metrics' exploration. - Properly encode label filters and query args before passing them to VictoriaMetrics. - Various cosmetic fixes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386	2022-12-22 17:17:58 -08:00
Yury Molodov	ec2b24f3f3	vmui: add explore tab for exploration of metrics, which belong to a particular job/instance (#3470 ) * feat: add "Explore" page * feat: add graphs for explore page * vmui: add explore tab for exploration of metrics, which belong to a particular job/instance * refactor: rename variables * refactor: extract graph to ExploreMetricItemGraph.tsx * feat: add searchable for Select.tsx * feat: improve metrics explorer * feat: set document title by page * feat: add page to view icons * fix: improve styles * fix: add encodeURIComponent to query	2022-12-22 17:17:47 -08:00
Aliaksandr Valialkin	3834602c9d	docs/CHANGELOG.md: document `1df87807fd` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3513	2022-12-22 15:22:21 -08:00
Yury Molodov	b3398a125f	vmui: step (#3521 ) * feat: add step rounding * fix: change step in URL parameters * refactor: change comment for roundStep	2022-12-22 15:18:04 -08:00
Aliaksandr Valialkin	c2fc996e01	lib/promscrape/discovery/azure: typo fix	2022-12-21 21:25:25 -08:00
Aliaksandr Valialkin	507e719c3f	app/vmselect/promql: add tests for `d3de110070`	2022-12-21 20:25:37 -08:00
Aliaksandr Valialkin	7888712185	lib/promrelabel: `make fmt` after `d3de110070`	2022-12-21 20:25:37 -08:00
Aliaksandr Valialkin	a77bb09e69	app/vmselect/promql: make sure that label_replace() doesn't create an empty dst_label if the src_label doesn't match regex	2022-12-21 20:25:37 -08:00
Aliaksandr Valialkin	cc482b89a3	lib/promrelabel: add support for `keepequal` and `dropequal` relabeling actions These actions are supported by Prometheus starting from v2.41.0 See https://github.com/prometheus/prometheus/pull/11564 , https://github.com/prometheus/prometheus/issues/11556 and https://github.com/prometheus/prometheus/issues/3756 Side note: It's a pity that Prometheus developers decided inventing `keepequal` and `dropequal` relabeling actions instead of adding support for `keep_if_equal` and `drop_if_equal` relabeling actions supported by VictoriaMetrics since June 2020 - see `2a39ba639d` .	2022-12-21 20:06:09 -08:00
Yury Molodov	6f17c1e866	vmui: fix change timezone (#3519 ) vmui: fix time picker with changed time zone	2022-12-21 13:08:42 -08:00
Aliaksandr Valialkin	fcee36081b	lib/bytesutil: make sure that the cleanup code is performed only by a single goroutine out of many concurrently running goroutines Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466	2022-12-21 12:58:27 -08:00
Aliaksandr Valialkin	ad010726fb	deployment/docker: update VictoriaMetrics tag from v1.85.2 to v1.85.3 in docker-compose files	2022-12-20 15:50:04 -08:00
Aliaksandr Valialkin	30b15111aa	docs/CHANGELOG.md: cut v1.85.3	2022-12-20 14:51:42 -08:00
Aliaksandr Valialkin	45a34dca4e	app/vmselect/vmui: `make vmui-update` after `731d189fa9`	2022-12-20 14:51:41 -08:00
Roman Khavronenko	998d89685f	vmagent: respect `-usePromCompatibleNaming` if no relabeling is set (#3511 ) * vmagent: respect `-usePromCompatibleNaming` if no relabeling is set See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: upd test Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-20 14:34:29 -08:00
Yury Molodov	88f408941c	fix: change the logic for hide query (#3514 )	2022-12-20 14:30:21 -08:00
Zakhar Bessarab	decf46d72b	app/vmbackupmanager: add metrics for better observability (#488 ) * app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response * app/vmbackupmanager: drop old metrics before creating new ones * app/vmbackupmanager: use `_total` postfix for counter metrics * app/vmbackupmanager: remove `_total` postfix for gauge-like metrics * app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention * app/vmbackupmanager: address review feedback * app/vmbackupmanager: fix metric name * app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage * app/vmbackupmanager: improve performance for backup size calculation * app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic * {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package * app/vmbackupmanager: fix creating new metrics instead of reusing existing ones * lit/formatutil: add comment to make linter happy * app/vmbackupmanager: address review feedback	2022-12-20 14:18:43 -08:00
Aliaksandr Valialkin	1ff62629f4	lib/storage: clear the err if it is set to io.EOF when searching for the TSID by metricID This is expected error after when recently added indexdb data isn't available for search yet or wasn't flushed to disk after unclean shutdown of VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515	2022-12-20 14:05:53 -08:00
Aliaksandr Valialkin	a6a2e35a83	Makefile: update golangci-lint version from v1.48.0 to v1.50.1	2022-12-20 13:10:42 -08:00
Aliaksandr Valialkin	2e714aa2d3	Makefile: publish release docker images at DockerHub with the `stable` tag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2911	2022-12-20 12:27:24 -08:00
Aliaksandr Valialkin	9d3c121feb	deployment/docker: use cluster-latest tag for docker images from cluster components Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3505	2022-12-20 10:38:51 -08:00
Zakhar Bessarab	09c1eab47d	app/vmbackupmanager: update doc to include cluster to cluster restore example (#3506 )	2022-12-20 10:23:03 -08:00
Roman Khavronenko	f2e7e11382	docs: fix link typo in operator docs (#3508 )	2022-12-20 10:22:40 -08:00
Aliaksandr Valialkin	874aea98b3	docs/CHANGELOG.md: document the change at 547f07463b29c09c62c9af35eac9cee6764b3286 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612	2022-12-20 10:21:22 -08:00
Aliaksandr Valialkin	276e867119	docs/CHANGELOG.md: formatting fix	2022-12-20 01:06:42 -08:00
Aliaksandr Valialkin	7fb314bb59	app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly Previously too short lookbehind window d for rate(m[d]) could be automatically extended if it didn't cover at least two raw samples. This was needed in order to guarantee non-empty results from rate(m[d]) on short time ranges. Now the lookbehind window isn't extended if it is set explicitly, since it is expected that the user knows what he is doing. The lookbehind window continues to be extended when needed if it isn't set explicitly. For example, in the case of rate(m). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483	2022-12-20 00:18:32 -08:00
Aliaksandr Valialkin	4c6c8c50b5	docs/Articles.md: add a link to https://www.youtube.com/watch?v=Mesc6JBFNhQ	2022-12-19 21:41:02 -08:00
Aliaksandr Valialkin	1ed0d07aa7	docs/CHANGELOG.md: add a warning for releases between v1.83.0 and v1.85.1 that it is recommended upgrading to v1.85.2 because of the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502	2022-12-19 13:37:20 -08:00
Aliaksandr Valialkin	cac56859eb	docs/CHANGELOG.md: consistently use YYYY-MM-DD format for release dates The previously used DD-MM-YYYY format could be confused with the MM-DD-YYYY format. The YYYY-MM-DD format reduces this confusion.	2022-12-19 13:33:09 -08:00
Aliaksandr Valialkin	2184de3bf2	lib/storage: do not check for the result returned by db.doExtDB() where this isn't necessary This simplifies the code a bit	2022-12-19 13:23:30 -08:00
Aliaksandr Valialkin	371f9085c9	docs/CHANGELOG.md: cut v1.85.2	2022-12-19 13:09:48 -08:00
Aliaksandr Valialkin	874a925310	vendor: `make vendor-update`	2022-12-19 13:07:52 -08:00
Aliaksandr Valialkin	9330da3195	lib/promscrape/discovery/consul: expose service tags in individual labels `__meta_consul_tag_<tagname>` This simplifies copying service tags to target labels with the following relabeling rule: - action: labelmap regex: __meta_consul_tag_(.+) See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus	2022-12-19 13:02:56 -08:00
Aliaksandr Valialkin	11bd290201	lib/storage: search for TSIDs for the given metricIDs in the previous indexdb if they aren't found in the current indexdb The issue triggers after the indexdb rotation for time series, which stop receiving new samples. This results in missing data for such time series in query responses. This commit should address the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502 The issue has been introduced in `2dd93449d8`	2022-12-19 11:56:49 -08:00
Aliaksandr Valialkin	8c08d625ee	lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid This should help improving performance at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-19 10:31:39 -08:00
Aliaksandr Valialkin	512c73cef9	lib/storage: properly set buf capacity inside marshalMetricID Previously it was always set to 0. In theory this could result into incorrect marshaling of metricIDs. The issue has been introduced in `5e4dfe50c6`	2022-12-19 10:31:38 -08:00
Aliaksandr Valialkin	2fad03d85e	lib/logger: follow-up for `72f8fce107` - Document the change at docs/CHANELOG.md - Log fatal errors if the -loggerJSONFields contains unexpected values - Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348	2022-12-16 17:44:05 -08:00
Michal Kralik	cffd2f79a1	lib/logger: support for renaming json fields (#3488 )	2022-12-16 17:43:39 -08:00
Aliaksandr Valialkin	70c720d640	lib/logger: follow-up for `72f8fce107` - Document the change at docs/CHANELOG.md - Log fatal errors if the -loggerJSONFields contains unexpected values - Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348	2022-12-16 17:42:31 -08:00
Aliaksandr Valialkin	2a229a319e	docs/vmalert.md: mention `latency_offset` query arg, which has been added in `86dae56bd0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481	2022-12-16 17:20:51 -08:00
Aliaksandr Valialkin	8d7d8f0125	app/vmselect/prometheus: follow-up after `86dae56bd0` Return error if the provided latency_offset query arg cannot be parsed. This should simplify debugging in production. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481	2022-12-16 17:16:26 -08:00
Roman Khavronenko	f8a3514e99	vmselect: support overriding of `-search.latencyOffset` (#3489 ) support overriding of `-search.latencyOffset` value via URL param `latency_offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-16 17:14:23 -08:00
Aliaksandr Valialkin	2e597555ec	lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page	2022-12-16 16:44:12 -08:00
Roman Khavronenko	9b82eebc3e	dashboards: respect $job var in sub-vars for cluster dash (#3487 ) Previously, $job_select, $job_storage and $job_insert didn't respect the $job filter. This change updates the variable queries to account for set $job variable. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-16 16:44:10 -08:00
Aliaksandr Valialkin	c8942f819f	docs/keyConcepts.md: update the list of supported data ingestion protocols - Add DataDog protocol - Remove native protocol, since it isn't intended for general-purpose usage by external clients	2022-12-15 12:02:51 -08:00
Aliaksandr Valialkin	fbeebe4869	lib/storage: skip missing tsids in the current block header by using binary search This improves performance by up to 10x when big number of the requested TSIDs are missing in the searched parts. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-14 22:07:55 -08:00
Aliaksandr Valialkin	1a88fe5b1f	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:29:57 -08:00
Aliaksandr Valialkin	3a28a52667	lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes	2022-12-14 17:53:18 -08:00
Aliaksandr Valialkin	f8e008f5ad	app/vmselect: show the link to /expand-with-exprs at /select/0/ page	2022-12-14 16:23:19 -08:00
Aliaksandr Valialkin	6d13ca2b88	app/vmselect: add /select/0/prometheus/expand-with-exprs page	2022-12-14 16:23:17 -08:00
Aliaksandr Valialkin	dafa819c54	.wwhrd.yml: add ISC license, which is used by github.com/davecgh/go-spew This license is compatible with Apache2 See https://github.com/davecgh/go-spew/blob/master/LICENSE	2022-12-14 14:56:06 -08:00
Aliaksandr Valialkin	ed7d204274	docs/vmagent.md: small formatting fix	2022-12-14 14:25:46 -08:00
Aliaksandr Valialkin	97a9d2eb04	docs/CHANGELOG.md: fix the link to the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466	2022-12-14 14:17:27 -08:00
Aliaksandr Valialkin	5a288e98bf	docs/CHANGELOG.md: add release date for v1.85.1	2022-12-14 14:12:35 -08:00
Aliaksandr Valialkin	5da63d1ae7	vendor: `make vendor-update`	2022-12-14 12:14:01 -08:00
Aliaksandr Valialkin	7f022e03f9	app/vmselect/vmui: `make vmui-update`	2022-12-14 12:02:05 -08:00
Aliaksandr Valialkin	e3675d3679	docs/CHANGELOG.md: cut v1.85.1	2022-12-14 11:57:57 -08:00
Aliaksandr Valialkin	7d1d82d094	docs/Cluster-VictoriaMetrics.md: mention the `vm_storage_is_read_only` metric, which can help debugging readonly mode at vmstorage	2022-12-14 09:32:21 -08:00
Aliaksandr Valialkin	cca16cb498	docs/vmagent.md: clarify that relabeling is actually a debugging at `relabel debug` section	2022-12-13 15:46:30 -08:00
Aliaksandr Valialkin	03ff3be993	docs/CHANGELOG.md: document the bugfix at `a50120a212`	2022-12-13 09:36:37 -08:00
Zakhar Bessarab	1e58eabde6	lib/backup/azremote: fix copying for parts larger than 256M by using async copy (#3479 ) * lib/backup/azremote: fix copying for parts larger than 256M by using async copy * lib/backup/azremote: add description of an error for log message	2022-12-13 09:36:36 -08:00
Yury Molodov	cf8ddc67ec	fix: prevent run query when selecting autocomplete option (#3480 )	2022-12-13 09:30:33 -08:00
Aliaksandr Valialkin	ea7940e5a7	lib/mergeset: reduce the parts threshold before starting assisted merges This should improve query speed in general case. This is a follow-up for `d1af6046c7`	2022-12-13 09:14:08 -08:00
Aliaksandr Valialkin	d66d28f917	app/vmselect/vmui: `make vmui-update`	2022-12-12 17:49:07 -08:00
Yury Molodov	664f8e0e12	vmui: minor enhancements (#3471 ) * update package-lock.json * fix: correct handle click by "action" on cardinality page * fix: correct styles for icons width * feat: add layout with copyright * feat: add website and issue to footer	2022-12-12 17:49:05 -08:00
Dima Lazerka	b410e53897	Add Anomaly Detection to enterprise features list (#3476 ) * Add Anomaly Detection to enterprise features list * Update docs/enterprise.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-12 17:43:16 -08:00
Aliaksandr Valialkin	2a190f6451	lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead Blocked small merges may result into big number of small parts, which, in turn, may result in increased CPU and memory usage during queries, since queries need to inspect all the existing small parts. The issue has been introduced in `8189770c50` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-12 17:01:33 -08:00
Aliaksandr Valialkin	f3e5c9c246	lib/bytesutil: cache results for all the input strings, which were passed during the last 5 minutes from FastStringMatcher.Match(), FastStringTransformer.Transform() and InternString() Previously only up to 100K results were cached. This could result in sub-optimal performance when more than 100K unique strings were actually used. For example, when the relabeling rule was applied to a million of unique Graphite metric names like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 This commit should reduce the long-term CPU usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 after all the unique Graphite metrics are registered in the FastStringMatcher.Transform() cache. It is expected that the number of unique strings, which are passed to FastStringMatcher.Match(), FastStringTransformer.Transform() and to InternString() during the last 5 minutes, is limited, so the function results fit memory. Otherwise OOM crash can occur. This should be the case for typical production workloads.	2022-12-12 14:47:00 -08:00
Zakhar Bessarab	600023a8f3	{app/{vmbackup/vmrestore}: update path example to use Azure terminology for consistency (#3475 )	2022-12-12 14:43:25 -08:00
Roman Khavronenko	a44af871d3	vmalert: support `$for` or `.For` template variables (#3474 ) support `$for` or `.For` template variables in alert's annotations. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-12 14:42:16 -08:00
Roman Khavronenko	4917c9ad8a	dashboards: add VersionChange annotation (#3473 ) The new annotation is hidden by default and suppose to show component `short_version` label change on the panels. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-12 14:41:46 -08:00
Aliaksandr Valialkin	87390443d1	lib/protoparser/datadog: do not re-use previously parsed field values if they are missing in the currently parsed message Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432	2022-12-11 13:09:41 -08:00
Aliaksandr Valialkin	87723ef0e1	vendor: update github.com/klauspost/compress from v1.15.12 to v1.15.13	2022-12-11 02:12:07 -08:00
Aliaksandr Valialkin	77aedb6008	docs/CHANGELOG.md: cut v1.85.0	2022-12-11 02:01:20 -08:00
Aliaksandr Valialkin	c135ea36c7	docs/CHANGELOG.md: document changes at v1.79.6	2022-12-11 01:51:48 -08:00
Aliaksandr Valialkin	2e1a44cfd0	docs/CHANGELOG.md: document `461158a437` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427	2022-12-10 23:42:29 -08:00
Aliaksandr Valialkin	7d996ced51	app/{vmagent,vminsert}/datadog: make the `host` label optional in DataDog data ingestion protocol Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432	2022-12-10 23:33:17 -08:00
Aliaksandr Valialkin	808ce815e4	app/vmselect/promql: allow passing `inf` arg into functions, which accept numeric limit on the number of output time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461	2022-12-10 22:48:01 -08:00
Aliaksandr Valialkin	afa232eebb	docs/CHANGELOG.md: add a link to the issue related to reduced CPU and memory usage at vmalert Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3464 The related commit - `b97bd01605`	2022-12-10 22:21:58 -08:00
Aliaksandr Valialkin	b472fbc737	vendor: `make vendor-update`	2022-12-10 21:47:18 -08:00
Aliaksandr Valialkin	00d953ab66	docs: clarify that single-node VictoriaMetrics also provides functionality for relabel debugging Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 20:50:51 -08:00
Aliaksandr Valialkin	a521135b7b	lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 12:47:47 -08:00
Aliaksandr Valialkin	72ea387c86	app/vmselect: properly serve static css for /select/.../metric-relabel-debug page	2022-12-10 03:04:43 -08:00
Aliaksandr Valialkin	97b41e727c	lib/promscrape: implement target-level and metric-level relabel debugging Target-level debugging is performed by clicking the 'debug' link at the corresponding target on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page. Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407 See https://docs.victoriametrics.com/vmagent.html#relabel-debug	2022-12-10 02:25:56 -08:00
Aliaksandr Valialkin	8fce069c7b	app/vmalert: properly handle nil req passed to requestToCurl() This fixes a panic in the TestAlertingRule_Exec_Negative test. The panic has been introduced in the commit `b97bd01605`	2022-12-10 02:05:20 -08:00
Aliaksandr Valialkin	7c311952aa	docs/url-examples.md: add missing whitespace for proper heading for the example on how to send data via OpenTSDB protocol	2022-12-09 17:33:55 -08:00
Aliaksandr Valialkin	ceed09877e	docs/CHANGELOG.md: document `b97bd01605`	2022-12-09 11:49:53 -08:00
Aliaksandr Valialkin	650d1d1ae5	app/vmalert: do not show system links at http://vmalert:8880/ page when it is requested via proxy The system links are absolute, e.g. they start from `/`, so there are high chances they won't work as expected when requested via proxy such as vmselect with -vmalert.proxyURL command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3424	2022-12-09 11:49:53 -08:00
Roman Khavronenko	385d082bca	vmalert: do not hold pointer to http.Request (#3467 ) http.Request was used as a part of state struct for generating the curl command when viewing the rule's state changes. It appears, that holding a referencing is far more expensive than generating the curl command immediately. On the test with 40k rules, this change reduces memory and CPU usage by 50%. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-09 11:49:53 -08:00
Aliaksandr Valialkin	04abd5e113	docs/CHANGELOG.md: document the bugfix at `05b42601c3` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247	2022-12-08 18:35:47 -08:00
Zakhar Bessarab	c939a8e8a2	lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403 ) * lib/promscrape/discovery/azure: remove API server from URL returned by azure * lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer	2022-12-08 18:35:46 -08:00
Aliaksandr Valialkin	6e390f3b99	lib/querytracer: fix remaining tests after `49ebc48809`	2022-12-08 18:18:50 -08:00
Aliaksandr Valialkin	e56d5e1918	lib/storage: follow-up after `7c0ae3a86a` - Update docs at https://docs.victoriametrics.com/#deduplication - Optimize the deduplication loop a bit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333	2022-12-08 18:18:49 -08:00
Roman Khavronenko	909cd04c55	lib/storage: keep sample with the biggest value on timestamp conflict (#3421 ) The change leaves raw sample with the biggest value for identical timestamps per each `-dedup.minScrapeInterval` discrete interval when the deduplication is enabled. ``` benchstat old.txt new.txt name old time/op new time/op delta DeduplicateSamples/minScrapeInterval=1s-10 817ns ± 2% 832ns ± 3% ~ (p=0.052 n=10+10) DeduplicateSamples/minScrapeInterval=2s-10 1.56µs ± 1% 2.12µs ± 0% +35.19% (p=0.000 n=9+7) DeduplicateSamples/minScrapeInterval=5s-10 1.32µs ± 3% 1.65µs ± 2% +25.57% (p=0.000 n=10+10) DeduplicateSamples/minScrapeInterval=10s-10 1.13µs ± 2% 1.50µs ± 1% +32.85% (p=0.000 n=10+10) name old speed new speed delta DeduplicateSamples/minScrapeInterval=1s-10 10.0GB/s ± 2% 9.9GB/s ± 3% ~ (p=0.052 n=10+10) DeduplicateSamples/minScrapeInterval=2s-10 5.24GB/s ± 1% 3.87GB/s ± 0% -26.03% (p=0.000 n=9+7) DeduplicateSamples/minScrapeInterval=5s-10 6.22GB/s ± 3% 4.96GB/s ± 2% -20.37% (p=0.000 n=10+10) DeduplicateSamples/minScrapeInterval=10s-10 7.28GB/s ± 2% 5.48GB/s ± 1% -24.74% (p=0.000 n=10+10) ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-08 18:18:36 -08:00
Aliaksandr Valialkin	323fc14e0a	lib/querytracer: fix tests after `49ebc48809`	2022-12-08 17:21:47 -08:00
Aliaksandr Valialkin	676de127aa	all: update Go builder from v1.19.3 to v1.19.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved	2022-12-08 17:04:41 -08:00
Aliaksandr Valialkin	a13f16e48a	lib/promscrape: allow using `sample_limit` and `series_limit` options in stream parsing mode Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 17:04:12 -08:00
Aliaksandr Valialkin	353aad9da9	app/vmselect/searchutils: do not print flag name responsible for query timeout if the timeout isn't reached This should make the log message more clear	2022-12-08 13:08:02 -08:00
Aliaksandr Valialkin	81da751e27	docs/Cluster-VictoriaMetrics.md: typo fix	2022-12-07 12:26:45 -08:00
Aliaksandr Valialkin	78582c3e82	docs/CHANGELOG.md: document the addition of file-based discovery of vmstorage nodes	2022-12-07 12:05:08 -08:00
Aliaksandr Valialkin	d009ba1def	docs/Cluster-VictoriaMetrics.md: `make docs-sync` after 5de8330ce00adfc5ac794070d30a2617ddc14bf2	2022-12-07 12:02:28 -08:00
Aliaksandr Valialkin	255c04bc20	lib/querytracer: put the version of VictoriaMetrics in the first message of query trace This should simplify further debugging, since the first thing to start the debugging by query trace is to know the version of VictoriaMetrics, which produced this trace.	2022-12-07 09:49:51 -08:00
Roman Khavronenko	5539beddb5	dashboards: remove DataLinks from single version (#3456 ) Those data links were copy&paste artifact from cluster version and aren't needed on the dash. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-07 09:49:32 -08:00
Roman Khavronenko	70c3664b40	alerts: remove `show_at` label for RequestErrorsToAPI alert (#3455 ) Alert `RequestErrorsToAPI` could be permanently triggered due to mistakes in clients configuration. However, such requests are unlikely to cause VM health state change. So there is no need in displaying this alert because there will be no correlation caused by it. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-07 09:48:59 -08:00
Denys Holius	5232775709	deployment/docker: bump Grafana version to v9.2.7 (#3454 ) see https://grafana.com/blog/2022/11/29/grafana-security-release-new-versions-with-high-severity-security-fix-for-cve-2022-31097/	2022-12-07 09:48:15 -08:00
Aliaksandr Valialkin	8162e4ee3f	docs: follow-up after e1bf2a85d0559d112908ce81597f3261d3a085c0 - Document the change at docs/CHANGELOG.md - Run `make docs-sync` for copying app/vmgateway/README.md to docs/vmgateway.md in order to propagate docs' changes to https://docs.victoriametrics.com/vmgateway.html	2022-12-06 21:05:38 -08:00
Aliaksandr Valialkin	a257c5fe53	app/vmselect: typo fixes in code comments	2022-12-06 20:59:17 -08:00
Aliaksandr Valialkin	6680a17fec	docs: document the addition of `-storageNode.discoveryInterval` command-line flag	2022-12-06 19:57:46 -08:00
Aliaksandr Valialkin	12307bc888	docs/Articles.md: change the link to `How we tried using VictoriaMetrics and Thanos at the same time` from Russian to English article	2022-12-06 16:31:12 -08:00
Roman Khavronenko	5bbb88902e	vmalert: correctly return error for RW failures (#3452 ) * vmalert: correctly return error for RW failures By mistake, in `0989649ad0` the error for remote write failures weren't return to user. This change fixes it. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-06 16:31:12 -08:00
Aliaksandr Valialkin	72ea7b90d5	docs/Articles.md: add a link to the talk "How do We Keep Metrics for a Long Time in VictoriaMetrics"	2022-12-06 00:55:47 -08:00
Aliaksandr Valialkin	9449b93281	docs/Articles.md: link to a video for the talk 'VictoriaMetrics: scaling to 100 million metrics per second'	2022-12-06 00:53:17 -08:00
Aliaksandr Valialkin	b6ffdde969	vendor: `make vendor-update`	2022-12-05 23:28:23 -08:00
Aliaksandr Valialkin	f9b199e9e9	docs/CHANGELOG.md: document `1e0666abb4`	2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin	6423c0b901	app/vmselect/vmui: `make vmui-update` after `7645d9ae00`	2022-12-05 23:10:42 -08:00
Pedro Gonçalves	84cfe4fcaf	Datadog - Add device as a tag if it's present as a field in the series object (#3431 ) * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:10:42 -08:00
Dima Lazerka	f98751bfc3	Fix nth-check version for css-select dep (#2666 ) Fixes security vulnerability in nth-check version <=1.0.2 My previous version pin was insufficient, as it was imported again through a different (svgo -> css-select).	2022-12-05 23:00:12 -08:00
Aliaksandr Valialkin	412508c81d	docs: follow-up for `7645d9ae00` - Document the change at docs/CHANGELOG.md - Document the feature at https://docs.victoriametrics.com/#vmui Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3446	2022-12-05 22:57:45 -08:00
Yury Molodov	99ac0a22ad	feat: add toggle query display by Ctrl (#3449 )	2022-12-05 22:57:33 -08:00
Yury Molodov	daaea87931	vmui: timezone select (#3414 ) * feat: add timezone selection * vmui: provide feature timezone select * fix: correct timezone with relative time Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-05 22:57:16 -08:00
Yury Molodov	08a68a2829	vmui: improve table view (#3377 ) * vmui: add compact table view (#3365) * feat: add compact table view * fix: add overflow table * fix: change table styles * vmui: compact table view * Update docs/CHANGELOG.md Co-authored-by: Michal Kralik <michal.kralik@percona.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-05 22:56:53 -08:00
Yury Molodov	8f501ca220	vmui: fix multi-line query (#3448 ) * fix: remove prevent nav by up/down keys for multi-line query * fix: add query params encode in URL	2022-12-05 22:53:31 -08:00
Roman Khavronenko	ff1c654006	deployment: update the README (#3447 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-05 22:06:16 -08:00
Aliaksandr Valialkin	98110a06d1	docs/CHANGELOG.md: document `fd43b5bad0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445	2022-12-05 22:02:16 -08:00
Aliaksandr Valialkin	c902500efc	app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q	2022-12-05 21:55:24 -08:00
Aliaksandr Valialkin	0a9992a9c6	lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown	2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin	e465cf5d57	docs/vmctl.md: `make docs-sync` after `86c31f2955` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2930	2022-12-05 17:24:37 -08:00
Zakhar Bessarab	c3276e57b4	app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450 )	2022-12-05 17:20:12 -08:00
Aliaksandr Valialkin	d2e34b8052	{dashboards,alerts}: subtitute `{type="indexdb"}` with `{type=~"indexdb.*"}` inside queries after `8189770c50` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-05 16:00:42 -08:00
Aliaksandr Valialkin	7d5c64eb7a	all: add `-inmemoryDataFlushInterval` command-line flag for controlling the frequency of saving in-memory data to disk The main purpose of this command-line flag is to increase the lifetime of low-end flash storage with the limited number of write operations it can perform. Such flash storage is usually installed on Raspberry PI or similar appliances. For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data. The in-memory data is searchable in the same way as the data stored on disk. VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal. The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin	020e3093a6	vendor: `make vendor-update`	2022-12-05 01:02:06 -08:00
Yury Molodov	c1f192610a	fix: add word-break for tooltip (#3437 )	2022-12-05 00:20:10 -08:00
Roman Khavronenko	73340dcb01	dashboards: add `Disk space usage %` and `Disk space usage % by type` panels (#3436 ) The new panels have been added to the vmstorage and drilldown rows. `Disk space usage %` is supposed to show disk space usage percentage. This panel is now also referred by `DiskRunsOutOfSpace` alerting rule. This panel has Drilldown option to show absolute values. `Disk space usage % by type` shows the relation between datapoints and indexdb size. It supposed to help identify cases when indexdb starts to take too much disk space. This panel has Drilldown option to show absolute values. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-05 00:19:36 -08:00
Roman Khavronenko	a922308438	vmalert: reduce allocations for Prometheus resp parse (#3435 ) Method `metrics()` now pre-allocates slices for labels and results from query responses. This reduces the number of allocations on the hot path for instant requests. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-05 00:18:11 -08:00
Aliaksandr Valialkin	9ac1174493	lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function	2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin	3a25a4b1de	app/{vminsert,vmselect}: speed up TestInitStopNodes()	2022-12-03 23:53:14 -08:00
Aliaksandr Valialkin	a13d21513e	lib/mergeset: panic when too long item is passed to Table.AddItems()	2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin	dccd70ce10	lib/storage: remove duplicate logging for filepath on errors	2022-12-03 23:15:28 -08:00
Aliaksandr Valialkin	813e8402f6	lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args	2022-12-03 23:10:26 -08:00
Aliaksandr Valialkin	bb93494eac	lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers This simplifies the code	2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin	106332cd9f	lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation	2022-12-03 23:03:30 -08:00
Aliaksandr Valialkin	ea55c16422	lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart This allows packing in-memory blocks with different compression levels depending on its contents. This may save memory usage.	2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin	eca7f32151	lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part Previously only data was compressed with the given compressLevel	2022-12-03 22:35:16 -08:00
Aliaksandr Valialkin	7ffa66d249	lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part This results in more correct reporting of memory usage for in-memory parts	2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin	886ce94739	lib/mergeset: reduce the time needed for the slowest tests	2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin	10a17bfa16	lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}	2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin	233301a549	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin	fd9d0a550b	lib/storage: remove logging redundant path values in a single error message	2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin	82f64072d2	lib/filestream: remove logging redundant path values in a single error message	2022-12-03 22:02:04 -08:00
Aliaksandr Valialkin	81400c80f0	lib/fs: remove logging redundant path values in a single error message	2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin	dc890ab80c	lib/backup: remove logging duplicate path values in a single error message	2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin	6910b1de2e	all: typo fix: `the the` -> `the`	2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin	0b8e7deabd	lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message appears in the log and then upgrade to newer release.	2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin	8e9822bc7f	lib/storage: speed up search for data block for the given tsids Use binary search instead of linear scan for looking up the needed data block inside index block. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin	df4eca2807	docs/CHANGELOG.md: consistently add `-` prefix in front of command-line flags This is a follow-up for `bcba5d2a78`	2022-12-02 19:14:02 -08:00
Aliaksandr Valialkin	86632658f0	docs: follow-up after `30fea30685` - Run `make docs-sync`, so app/vmalert/README.md is copied to docs/vmalert.md - Clarify the feature description in the docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3408	2022-12-02 19:14:02 -08:00
Denys Holius	5cbe8101c8	docs/Articles.md: fir broken link (#3433 )	2022-12-02 19:12:46 -08:00
Roman Khavronenko	2d8ac5355b	docs: fix typo in cluster's README (#3430 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-02 19:11:45 -08:00
Zakhar Bessarab	de0e4f1f7e	make: make openssl output parsing symbol number agnostic (#3429 )	2022-12-02 19:10:12 -08:00
Roman Khavronenko	31ca22109e	vmalert: fix replay step param (#3428 ) The recent change in modifying default value of `datasource.queryStep` flag resulted in situation where replay mode was always running queries with step=`datasource.queryStep`. When it should always use rule's evaluation interval. The fix is related not to replay mode only, but for all Range requests. Now step param is set individually for each mode. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-02 19:09:30 -08:00
Roman Khavronenko	51993e7d3e	dashboards: fix typo in data link (#3426 ) Fixes a missing `&` char in data link for ETA panel on cluster dashboards. Without `&` char it generates wrong link when click on Drilldown menu. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-02 19:05:42 -08:00
Zakhar Bessarab	59f889cd3f	app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url` (#3423 ) * app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url` * vmalert: remove WriteTimeout from clients Cfg No need to have it as a part of configuration struct: * the client isn't used by other packages; * there are no internal tests to check the WriteTimeout. * vmalert: remove DisablePathAppend from clients Cfg No need to have it as a part of configuration struct: * the client isn't used by other packages; * there are no internal tests to check the DisablePathAppend. Co-authored-by: hagen1778 <roman@victoriametrics.com>	2022-12-02 19:03:34 -08:00
Roman Khavronenko	435f6f3add	vmalert: properly pass headers during the restore procedure (#3420 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-02 18:53:44 -08:00
Aliaksandr Valialkin	d8303845ef	lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC	2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin	d8d4d21d7a	lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts Move the common code into releasePartsToMerge() method and consistently use it throughout the code.	2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin	50e6a137bb	docs/vmagent.md: update after `959f06d175`	2022-11-29 21:34:00 -08:00
Aliaksandr Valialkin	be6da5053f	lib/promscrape: optimize service discovery speed - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin	8440c822fa	all: follow-up after `05cf8a6ecc`	2022-11-29 21:11:42 -08:00
Aliaksandr Valialkin	2a107cc8a7	app/vmalert: substitute -datasource.disablePathAppend with -remoteRead.disablePathAppend in the description for -datasource.url command-line flag This is a follow-up for `959f06d175`	2022-11-29 21:11:18 -08:00
Aliaksandr Valialkin	2524d94fe1	lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels	2022-11-29 21:11:03 -08:00
Dmytro Kozlov	002c028f22	vmctl: support of the remote read protocol (#3232 ) vmctl: support of the remote read protocol Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2022-11-29 21:08:47 -08:00
Roman Khavronenko	b2f45b4856	dashboards: update VM single dash (#3400 ) The change list is the following: * bump Grafana version to 9.2.6; * replace old "Graph" panel with "TimeSeries" panel; * show % usage of Mem and CPU additionally to of absolute values; * `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`; * add Annotations for Alert triggers. Not all alerts are supposed to be displayed on the dashboard, but only those with label `show_at: dashboard`. See `alerts.yml` change. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-29 20:39:05 -08:00
Roman Khavronenko	e42308e84c	dashboards: update vmalert dash (#3404 ) The change list is the following: * bump Grafana version to 9.2.6; * replace old Graph panel with TimeSeries panel; * add RemoteWrite section; * allow configuring topK elements for some of the panels; * Preer grouping by job instead of grouping by instance. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-29 20:38:35 -08:00
Max Golionko	d272a8270b	vmalert: flag reference update (#3415 ) * flag reference update there is no flag `-datasource.disablePathAppend` and datasource actually checking for `-remoteRead.disablePathAppend` * update source for doc as well	2022-11-29 20:38:02 -08:00
Roman Khavronenko	e6f0da480e	dashboards: update vmagent dash (#3411 ) The change list is the following: * bump Grafana version to 9.2.6; * add version change annotations; * switch to per-job panels instead of per-instance; * add drilldown option for resource usage panels. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-29 20:29:34 -08:00
Zakhar Bessarab	5edaff5165	doc/operator: update formatting for backup section, add FAQ section (#3416 ) * doc/operator: update formatting for backup section, add FAQ section * doc/operator: address review feedback * doc/operator: add note about difference between `VMRestore` and `VMBackupmanager` as init containers * Update docs/operator/backups.MD Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-11-29 20:28:50 -08:00
Aliaksandr Valialkin	a547226b53	docs/CHANGELOG.md: document `027ab74efb` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402	2022-11-28 18:55:17 -08:00
Aliaksandr Valialkin	027ab74efb	lib/httpserver: link to url format docs in error message emitted on parse error for the provided url path This should help users identifying and fixing improperly set up urls. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402	2022-11-28 18:49:03 -08:00
Aliaksandr Valialkin	20ceb3a2ca	docs/url-examples.md: allow linking to `how to send OpenTSDB/Graphite data` chapters	2022-11-28 18:42:56 -08:00
Aliaksandr Valialkin	8ce5b095b7	lib/promscrape: add `exported_` prefix to metric names exported by scrape targets if they clash with automatically generated metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:34 -08:00
Aliaksandr Valialkin	ab96fed372	docs/Articles.md: add a link to slides about scaling to 100 million metrics per second	2022-11-28 16:43:58 -08:00
Roman Khavronenko	85d0cbbfc6	dashboards: update VM cluster dash (#3401 ) The change list is the following: * bump Grafana version to 9.2.6; * remove artifacts in data links. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-28 16:43:58 -08:00
Denys Holius	a90fbafa23	deployment/docker: bump grafana version to latest v9.2.6 (#3398 )	2022-11-28 16:43:58 -08:00
Timur Bakeyev	b6064dd645	Update `datasource` entries consistently contain type `prometheus` and uid `$ds`. (#3393 ) Co-authored-by: Timour I. Bakeev <tbakeev@ripe.net>	2022-11-28 16:43:58 -08:00
匠心零度	d4808d5b84	lib/storage: remove extra error check (#3396 )	2022-11-28 17:07:11 +01:00
Aliaksandr Valialkin	92a7f71ee1	deployment/docker: update VictoriaMetrics tag from v1.83.1 to v1.84.0	2022-11-25 22:29:20 -08:00
Aliaksandr Valialkin	d0461bec25	docs/CHANGELOG.md: cut v1.84.0	2022-11-25 19:59:30 -08:00
Aliaksandr Valialkin	154de24e60	vendor: `make vendor-update`	2022-11-25 19:53:38 -08:00
Aliaksandr Valialkin	94ab66e0a4	docs/README.md: `make docs-sync` after `58d459e8a8`	2022-11-25 16:56:04 -08:00
Aliaksandr Valialkin	c5eebaffd8	app/{vminsert,vmagent}: follow-up after `53a63c6c4c` Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:42:38 -08:00
Pedro Gonçalves	47a0d845c4	Adding pushgateway basic capabilities to vmagent (#3360 ) * init pushgateway implementation * Initial implementation of pushgateway in vmagent * Initial implementation of pushgateway in vmagent	2022-11-25 16:37:28 -08:00
Zakhar Bessarab	e407e7243a	{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348 ) * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * app/vmselect: fix error message * {app/vmstorage,app/vmselect}: fix error messages * app/vmselect: change log level for error handling * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-25 10:32:45 -08:00
Aliaksandr Valialkin	693d68f307	app/vmselect/vmui: `make vmui-update` after `37cda9abd0`	2022-11-25 07:34:03 -08:00
Yury Molodov	fc8ff75e27	fix: change header settings (#3391 )	2022-11-25 07:28:24 -08:00
Yury Molodov	805d93dfec	refactor: create Autocomplete component (#3390 )	2022-11-25 07:28:24 -08:00
Roman Khavronenko	ed39d0d11c	dashboards: cleanup & remove artifacts (#3387 ) * some unexpected DS UIDs were removed; * replace `$instance.` filter with `$instance` since we respect the instance port anyway; remove predefined datasource for `clusterbytenant` in favour of datasource variable `ds`. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-25 07:28:24 -08:00
Aliaksandr Valialkin	2f7152deb3	app/vmselect/vmui: `make vmui-update` after `eb772aa50e`	2022-11-24 17:37:33 -08:00
Aliaksandr Valialkin	6b2c85d6bf	docs/MetricsQL.md: document that `histogram_share()` accepts optional `boundsLabel` arg	2022-11-24 17:27:43 -08:00
Aliaksandr Valialkin	c7bd0f5082	docs/vmagent.md: typo fix	2022-11-24 17:19:56 -08:00
Roman Khavronenko	cae148d5c6	dashboards: cluster dashboard update (#3380 ) The purpose of the update is to make the dash more usable for large installations with many instances. Panels which showed metrics per-instance (Mem, CPU) now are showing metrics per-job or min/max/avg aggregations in % instead. This supposed to help immediately to identify resource shortage and remain usable for small and big installations. For cases when detailed info is needed, to the bottom of the dashboard a new row `Drilldown` was added. Panels like Mem or CPU now contain a `data-link` named `Drilldown` (cis shown on line click) which takes user to more detailed panel. The change list is the following: * bump Grafana version to 9.1.0; * replace old "Graph" panel with "TimeSeries" panel; * improve Uptime panel to show number of instances per job; * show % usage of Mem and CPU instead of absolute values; * `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`; * add `Drilldown` section for detailed resource usage; * add Annotations for Alert triggers. Not all alerts are supposed to be displayed on the dashboard, but only those with label `show_at: dashboard`. See `alerts-cluster.yml` change. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-24 13:20:10 -08:00
Aliaksandr Valialkin	a5e31c9c23	docs/Articles.md: add a link to "The cost of scale in Prometheus ecosystem" talk	2022-11-23 00:11:42 +02:00
Aliaksandr Valialkin	587f90b4cb	app/vmselect/vmui: `make vmui-update` after `7dc2349913`	2022-11-22 15:36:19 +02:00
Yury Molodov	a59c7a0dd7	vmui: add set up series custom limits (#3368 ) * feat: add set up series custom limits * feat: add button for show series without limits * fix: resolve merge conflicts	2022-11-22 15:36:17 +02:00
Aliaksandr Valialkin	e45022c344	vendor: `make vendor-update`	2022-11-22 11:26:34 +02:00
Aliaksandr Valialkin	8f53c41aa1	docs/Articles.md: add a link to https://www.youtube.com/watch?v=_zORxrgLtec (OSA Con 2022: Specifics of data analysis in Time Series Databases)	2022-11-22 01:08:12 +02:00
Aliaksandr Valialkin	afc35485c1	app/vminsert: add missing vm_relabel_config_* metrics after `03d88bc066` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345	2022-11-22 00:48:13 +02:00
Roman Khavronenko	d1169c1559	vmagent: expose metrics for tracking config state (#3375 ) Expose `vm_relabel_config_` and `vm_promscrape_config_` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin	2f9df62795	app/vmselect/vmui: `make vmui-update` after `7d1b3e7e14`	2022-11-22 00:35:05 +02:00
Yury Molodov	5924aaf7e5	vmui: add copy button to row on Table view (#3363 ) * feat: add copy button to row on Table view * vmui: add copy button to row on Table view Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-22 00:35:05 +02:00
Yury Molodov	8bc54b6d18	vmui: minor fixes (#3361 ) * fix: reset the value of the switches trace and cache * fix: add cursor text for inputs * fix: solve the Infinite loop of useFetchQuery.ts * fix: change condition for show/hide autocomplete * fix: add limit error length for input	2022-11-22 00:35:05 +02:00
Yury Molodov	05712cfc8d	vmui: sticky tooltip (#3376 ) * feat: add ability to make tooltip "sticky" * vmui: add ability to make tooltip "sticky"	2022-11-22 00:35:05 +02:00
Aliaksandr Valialkin	5c65b3c7dc	docs/CHANGELOG.md: link to the related issue for range_normalize() function	2022-11-21 23:27:11 +02:00
Aliaksandr Valialkin	ad548abd6a	app/vmselect/promql: add `range_normalize(q1, ..., qN)` function for normalizing query results into [0..1] value range This may be useful for analyzing correlation between time series with different value ranges	2022-11-21 23:25:35 +02:00
Aliaksandr Valialkin	c33bcae457	lib/promscrape/discovery/gce: do not pass filter arg when discovering zones The filter arg isn't supported by zones API in GCE. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202	2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin	9cb76c9cd3	app/vmselect/promql: allow using SI and IEC suffixes in numeric values inside queries For example, 10Ki is equivalent to 101024, while 5.3M is equivalent to 5.31000*1000	2022-11-21 21:28:34 +02:00
Aliaksandr Valialkin	2d361a834d	app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312	2022-11-21 16:48:10 +02:00
Roman Khavronenko	67c60dbf86	vmalert: bump alerting rules evaluation interval to reasonable 30s (#3374 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-21 16:48:09 +02:00
Aliaksandr Valialkin	b0fefe562a	app/vmselect/promql: optimize `e1 op e2` when `e1` returns an empty result Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349	2022-11-21 16:09:49 +02:00
Roman Khavronenko	0475f8a38e	vmalert: add default list of alerting rules (#3373 ) The default list of alerting rules contains the basic rules for checking vmalert's health state and is recommended to use for monitoring vmalert deployments. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-21 16:09:47 +02:00
Aliaksandr Valialkin	373550b21a	README.md: sync with enterprise-cluster branch after `0e51967970`	2022-11-21 15:46:28 +02:00
Aliaksandr Valialkin	6fe8eec745	all: add a link to https://docs.victoriametrics.com/enterprise.html into description for enterprise flags	2022-11-21 15:44:54 +02:00
Aliaksandr Valialkin	0e51967970	app/{vminsert,vmselect}: add `-storageNode.filter` command-line flag for filtering the discovered storage nodes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353	2022-11-21 15:20:20 +02:00
Aliaksandr Valialkin	353b137ff0	app/vmselect/netstorage: typo fix after `61736e4a1d`	2022-11-19 00:53:54 +02:00
Aliaksandr Valialkin	97eafbe4a7	app/vmselect: clarify that it isnt recommended setting -replicationFactor at vmselect nodes even if the replication is enabled at vminsert nodes	2022-11-18 14:04:12 +02:00
Aliaksandr Valialkin	61736e4a1d	app/vmselect/netstorage: remove superflouos map lookup at ProcessSearchQuery This should reduce CPU usage a bit during querying	2022-11-18 13:49:59 +02:00
Aliaksandr Valialkin	eb784ff399	app/vmselect/netstorage: emit more useful information in query traces when some of vmstorage nodes return errors or if there is no need to wait for their responses	2022-11-18 13:01:42 +02:00
Yury Molodov	60aaf4b2c2	vmui: add trace analyzer (#3310 ) * refactor: change structure project * refactor: change structure project * fix: add hooks for set query params * refactor: add index for pages * docs: add TESTCASES.md * refactor: restructure components * feat: add page with trace analyzer * fix: change detect trace data * Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * fix: change descriptions on trace page * Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * feat: add base components * feat: add reset styles * docs: add description about trace analyzer * feat: add styles for custom panel page * feat: add styles for predefined panels * feat: add style for TracingsView.tsx * feat: add Alerts * feat: add Tooltip.tsx * fix: correct styles * feat: add DatePicker.tsx * feat: add tables * feat: add theme provider * fix: replace using callbacks as props to handlers * fix: correct update time * fix: change TimePicker.tsx * fix: correct styles * fix: update packages * vmui: refactor code, remove material-ui * feat: add paste json for trace analyzer * vmui: update trace analyzer docs * app/vmselect/vmui: `make vmui-update` Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-17 22:23:05 +02:00
Aliaksandr Valialkin	f96bf852ea	docs/CHANGELOG.md: document the fix for CPU usage spikes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-17 22:03:03 +02:00
Aliaksandr Valialkin	48dda17e36	lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes. This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-17 21:55:11 +02:00
Aliaksandr Valialkin	7969ca6c80	app/vmselect/vmui: `make vmui-update` after `51bfd1ab80`	2022-11-17 18:57:44 +02:00
Yury Molodov	734172d2d0	vmui: add ability hide query (#3359 ) * feat: add ability hide query * fix: change logic hide query * fix: remove console.log	2022-11-17 18:57:42 +02:00
Aliaksandr Valialkin	478a295208	vendor: `make vendor-update`	2022-11-17 01:38:59 +02:00
Dmytro Kozlov	6f7956503f	app/vmstorage: fix potential file inclusion via variable (#3339 ) * app/vmstorage: fix potential file inclusion via variable * app/vmstorage: cleanup	2022-11-17 01:31:37 +02:00
Aliaksandr Valialkin	f67c2a2f8f	app/vmselect/vmui: `make vmui-update` after `bc8a782f74`	2022-11-17 01:21:56 +02:00
Yury Molodov	423eb73051	vmui/refactor (#3298 ) * refactor: change structure project * refactor: change structure project * fix: add hooks for set query params * refactor: add index for pages * docs: add TESTCASES.md * refactor: restructure components * feat: add base components * feat: add reset styles * feat: add styles for custom panel page * feat: add styles for predefined panels * feat: add style for TracingsView.tsx * feat: add Alerts * feat: add Tooltip.tsx * fix: correct styles * feat: add DatePicker.tsx * feat: add tables * feat: add theme provider * fix: replace using callbacks as props to handlers * fix: correct update time * fix: change TimePicker.tsx * fix: correct styles * fix: update packages * vmui: refactor code, remove material-ui Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-17 01:21:54 +02:00
Aliaksandr Valialkin	0cc59b9f95	app/vmselect/promql: add `range_stdvar()` and `range_stddev()` functions for calculating variance and deviation over time series on the selected time range	2022-11-17 01:03:02 +02:00
Aliaksandr Valialkin	facf7efa3a	app/vmselect/promql: add `range_linear_regression(q)` function for calculating simple linear regression for the selected time series on the selected time range	2022-11-17 00:39:17 +02:00
Aliaksandr Valialkin	047fe3ee67	lib/promscrape: add a benchmark for internLabelStrings()	2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin	b0b8f05fa4	lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() The issue has been introduced in `58b40f514c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin	c8d7e1312c	lib/workingsetcache: add `-prevCacheRemovalPercent` command-line flag for tuning memory usage vs CPU usage ratio Reduce the default value of this flag from 1% to 0.2% after `71335e6024` This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:41:37 +02:00
Aliaksandr Valialkin	06300fe9b8	lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf Previously indexBlock.bhs pointed to the buffer, which could be changed over time. This could result in incorrect time series search over time. This is a follow-up for `58b40f514c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin	454060fd78	lib/mergeset: remove string allocation and copying when unmarshaling blockHeader This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin	6e4c9d5f6b	docs/CHANGELOG.md: document changes in v1.79.5 release	2022-11-11 01:28:12 +02:00
Aliaksandr Valialkin	513257d536	deployment/docker: update VictoriaMetrics version from v1.83.0 to v1.83.1	2022-11-11 01:25:00 +02:00
Aliaksandr Valialkin	6d984cdeff	docs/CHANGELOG.md: cut v1.83.1	2022-11-10 14:07:20 +02:00
Aliaksandr Valialkin	3bdd1727b7	docs/CHANGELOG.md: document the fix at `71335e6024`	2022-11-10 13:47:32 +02:00
Aliaksandr Valialkin	372b449f94	vendor: `make vendor-update`	2022-11-10 13:47:30 +02:00
Dmytro Kozlov	8b6bfae5ca	vmui: fix vmui vulnerability (#3336 ) * vmui: fix vmui vulnerability * vmui: code cleanup	2022-11-10 13:32:50 +02:00
Aliaksandr Valialkin	a5a0fa75bf	lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1% It has been appeared that some production workloads could suffer for some time after every reset of the previous cache when it gets less than 5% of requests after the needed item isn't found in the current cache. This could result in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM usage needed for reading, unpacking and caching the missed data from disk. This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%. This should reduce the possible negative impact after each cache reset by at least 5x, while reducing the total memory used by caches. This is a follow-up for `d906d8573e`	2022-11-10 12:38:53 +02:00
Aliaksandr Valialkin	58e7963828	app/vmselect/vmui: `make vmui-update` after `7130af7fd2`	2022-11-09 16:43:09 +02:00
Aliaksandr Valialkin	24213eaeba	lib/promscrape: add more cases to TestAddRowToTimeseries This is a follow-up for `16fdd2af8a`	2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL	87375b004a	test(golang): add test to function addRowToTimeseries (#3282 ) Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>	2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin	2091693f16	lib/protoparser/opentsdb: follow-up after `04b0e4e7bf` - Simplify the parser code to be less error prone - Document the change - Add a test for OpenTSDB put line with trailing whitespace without tags Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290	2022-11-09 15:36:15 +02:00
Roman Khavronenko	71dfe4d697	protoparser/opentsdb: allow lines without tags (#3303 ) According to http://opentsdb.net/docs/build/html/api_telnet/put.html "At least one tag pair must be present". However, in VictoriaMetrics datamodel tags aren't required. This could be confusing for users. Allowing accept lines without tags seems to do no harm. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin	28e4364fd2	docs/CHANGELOG.md: document `7130af7fd2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2814	2022-11-09 12:16:23 +02:00
Michal Kralik	051aa079cb	vmui: show tracing in json view (#3316 ) * vmui: show tracing in json view * vmui: refactor tracing view	2022-11-09 12:13:05 +02:00
dependabot[bot]	9c5d01d8c3	build(deps): bump loader-utils in /app/vmui/packages/vmui (#3328 ) Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.2 to 2.0.3. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.3/CHANGELOG.md) - [Commits](https://github.com/webpack/loader-utils/compare/v2.0.2...v2.0.3) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-11-09 12:11:52 +02:00
Denys Holius	c09cee96ca	guides/README.md: fix link to guide-delete-or-replace-metrics.html (#3331 )	2022-11-09 12:01:23 +02:00
Aliaksandr Valialkin	001b4da314	app/vmui/packages/vmui: return back accidental changes at `9f8bf524ad`	2022-11-09 11:58:13 +02:00
Roman Khavronenko	8ee464b22b	bump go version to 1.19.3 (#3327 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 11:56:38 +02:00
Michal Kralik	651a3aa18b	vmui: change graph legend label format (#3315 )	2022-11-09 11:52:48 +02:00
Zakhar Bessarab	e188b59382	docs/operator: fix description for SA at VMClusterSpec (#3313 )	2022-11-09 11:51:47 +02:00
Aliaksandr Valialkin	fe8d40f12c	app/{vminsert,vmselect}: test initialization with different number of storage nodes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329	2022-11-09 11:48:39 +02:00
Aliaksandr Valialkin	8540dd669b	app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket This should prevent from panics when the list of discovered vmstorage nodes changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329	2022-11-09 11:45:24 +02:00
Aliaksandr Valialkin	abf7e4e72f	lib/promscrape/discovery/consul: add `__meta_consul_partition` label in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/11482	2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin	9108a1d33f	vendor: update github.com/urfave/cli/v2 from v2.23.2 to v2.23.4	2022-11-07 14:59:05 +02:00
Aliaksandr Valialkin	d3035b1ca1	lib/storage: follow-up for `790768f20b` - Document the bugfix at docs/CHANGELOG.md - Simplify the bugfix a bit	2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin	be78950011	lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8	2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin	9d901ee55a	lib/envtemplate: allow non-env var names inside "%{ ... }"	2022-11-07 13:16:00 +02:00
Aliaksandr Valialkin	99e6a937a5	lib/storage: remove unused isFull field from hourMetricIDs struct	2022-11-07 13:15:59 +02:00
Aliaksandr Valialkin	4c2963e015	vendor: `make vendor-update`	2022-11-05 10:40:30 +02:00
Aliaksandr Valialkin	61bcbfd697	deployment/docker: update Go builder from v1.19.2 to v1.19.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved	2022-11-05 10:19:24 +02:00
Aliaksandr Valialkin	68382fe580	docs/Single-server-VictoriaMetrics.md: mention about security certifications at Security chapter This is a follow-up for `94bd49402e`	2022-11-05 10:09:38 +02:00
Artem Navoiev	6cd36a7fc3	docs: Add link to security page from Readme (#3286 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-11-05 10:09:38 +02:00
Dmytro Kozlov	ae05b475d0	docs/operator: change VMAgentRemoteWriteSettings.MaxDiskUsagePerURL type from int32 to int64 (#3307 ) * docs/operator: change VMAgentRemoteWriteSettings.MaxDiskUsagePerURL type from int32 to int64 * docs: operator updates api description Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-11-05 10:09:38 +02:00
Aliaksandr Valialkin	56ef9f2d88	docs/guides/guide-vmcluster-multiple-retention-setup.md: clarify docs after `a75d85b11e`	2022-11-05 10:09:11 +02:00
Roman Khavronenko	803ab6ed30	vmctl: fix panic on start (#3300 ) The change disables initing the `-version` flag in new `urfave/cli/v2` update. The `-version` flag conflicts with the identical flag from `lib/buildinfo` and causes panic. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3299 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-05 10:09:11 +02:00
Artem Navoiev	b030662e3e	update multi-tenancy guide. Add infromation about Enterprise and Rete… (#3285 ) docs: update multi-tenancy guide Add information about Enterprise and Retention Filters Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-11-05 10:09:11 +02:00
Zakhar Bessarab	fd36c9b3b1	docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr` (#3314 ) docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`	2022-11-04 09:49:53 +01:00
Zakhar Bessarab	7afa67dba0	{app/vmselect,app/vminsert}: ensure `-storageNodes` flag has no empty values (#3291 )	2022-11-01 14:54:55 +02:00
Aliaksandr Valialkin	be0aaa1e93	vendor: update github.com/urfave/cli/v2 from 2.20.3 to 2.23.0	2022-11-01 10:53:19 +02:00
Aliaksandr Valialkin	45fa0adeb7	docs/Single-server-VictoriaMetrics.md: follow-up after `a1a97b9321` - Remove trailing whitespace at the end of lines - Remove redundant sentence stating that time series matching the given selector will be deleted. It should be clear from the surrounding context.	2022-11-01 10:53:19 +02:00
Dmytro Kozlov	03c55fa646	docs: clarify information about usage of the `api/v1/admin/tsdb/delete_series` API (#3287 ) docs: clarify information about usage of the `api/v1/admin/tsdb/delete_series` API	2022-11-01 10:53:19 +02:00
Aliaksandr Valialkin	616c9f1fb0	docs/Release-Guide.md: instruct to update VictoriaMetrics version in deployment/docker/docker-compose*.yml files after creating new release This is a follow-up for `d1509f4559`	2022-11-01 10:32:02 +02:00
Denys Holius	63c7a84ab9	docker-compose: bump version of container tags for VictoriaMetrics components (#3294 ) * deployment/docker/docker-compose-cluster.yml: bump VictoriaMetrics Cluster components to the latest v1.83.0 version * deployment/docker/docker-compose.yml: bump VictoriaMetrics Single node and vmutils to the latest v1.83.0 version	2022-11-01 10:32:02 +02:00
Aliaksandr Valialkin	426a9b3bb1	docs/enterprise.md: mention that feature requests from enterprise customers are prioritized	2022-11-01 10:27:35 +02:00
Aliaksandr Valialkin	4a6d5ab1b1	lib/promrelabel: go fmt after `5cec9706dc`	2022-10-29 05:17:49 +03:00
Aliaksandr Valialkin	ac7c43c246	docs: typo fixes	2022-10-29 04:53:03 +03:00
Aliaksandr Valialkin	a72bf87e04	lib/promrelabel: add a test from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251	2022-10-29 04:34:08 +03:00
Aliaksandr Valialkin	1357107a11	docs/CHANGELOG.md: cut v1.83.0	2022-10-29 02:55:08 +03:00
Aliaksandr Valialkin	b259c55c1b	vendor: `make vendor-update`	2022-10-29 02:54:08 +03:00
Aliaksandr Valialkin	cf5efb93d9	app/vmbackupmanager: add functionality for automated restore from backup	2022-10-29 02:31:06 +03:00
Aliaksandr Valialkin	eae334f70e	lib/envflag: small refactoring after `518c340ae3` and `02096e06d0`	2022-10-29 02:29:19 +03:00
Aliaksandr Valialkin	ac5528cb46	lib/promscrape: properly add `exported_` prefix to labels, which clash with target labels if `honor_labels: true` option isn't set. The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function. The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_` prefix to labels inside `labels` slice become invisible in the returned `dst` labels. While at it, properly handle some corner cases: - Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix. - Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278 Thanks to @jplanckeel for the initial attempt to fix this issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281	2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin	ba739c8052	lib/promscrape/discovery/kubernetes: do not print an empty `kubeconfig_file` option in yaml at `/config` page	2022-10-28 22:15:30 +03:00
Aliaksandr Valialkin	a02a5f646f	app/vmselect/vmui: `make vmui-update` after `54e1865d17`	2022-10-28 14:52:16 +03:00
Yury Molodov	9732790935	vmui: minor fixes (#3276 ) * feat: apply serverURL on down Enter * fix: change method of set time range * fix: remove prevent run fetch without changes * fix: prevent reset timerange when autorefresh	2022-10-28 14:52:15 +03:00
Aliaksandr Valialkin	ed1056d3c9	docs/Cluster-VictoriaMetrics.md: improve docs for `dns+srv` service discovery	2022-10-28 14:23:02 +03:00
Aliaksandr Valialkin	09cf136154	app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446	2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin	976bbe3677	app/{vminsert,vmselect}: limit the access to storageNodes to getStorageNodesBucket and setStorageNodesBucket functions This makes the code more maintainable and earier to test.	2022-10-28 11:41:55 +03:00
Aliaksandr Valialkin	1386b6bd69	vendor: `make vendor-update`	2022-10-28 00:19:05 +03:00
Aliaksandr Valialkin	7ae038766c	app/vmalert/templates: properly escape all the special chars in `quotesEscape` function Previously the `quotesEscape` function was escaping only double quotes. This wasn't enough, since the input string could contain other special chars, which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r), backslash char, etc. This led to the following issues, which were improperly fixed: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue was "fixed" by introducing the `crlfEscape` function, which led to unnecessary complications in user templates, while not fixing various corner cases such as backslash chars in the input string. See `1de15ad490` - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue was "fixed" by urlencoding the whole string passed to -external.alert.source command-line flag. This led to invalid urls, which couldn't be parsed by Grafana. See `00c838353d` and `4bd0244599` This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings. This commit deprecates crlfEscape template function and adds the following new template functions: - strvalue and stripDomain - these functions are supported by Prometheus, so they were added for compatibility purposes. - jsonEscape and htmlEscape for converting the input string to valid quoted JSON string and for html-escaping the input string, so it could be safely embedded as a plaintext into html. This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.	2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin	8a6898b625	Revert "vmalert: escape query params if external alert source defined (#3267 )" This reverts commit `00c838353d`. Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 . Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana: https://grafana.example.com/explore%3ForgId%3D1%26left%3D... The next commit will contain the correct fix of the issue - the `quotesEscape` function must properly escape the string, so it could be embedded into JSON string. This function must properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary. Actually, the next commit makes the `crlfEscape` function deprecated.	2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin	f4e9393076	vendor: `make vendor-update`	2022-10-28 00:08:50 +03:00
Denys Holius	6e5308ecb2	.github/workflows/codeql-analysis.yml: specifically setting the Go version (#3277 ) see https://github.com/github/codeql-action/issues/1059	2022-10-28 00:08:50 +03:00
Dmytro Kozlov	3123059407	vmalert: escape query params if external alert source defined (#3267 ) vmalert: escape query args if external alert source defined	2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin	450a32970a	lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax This is a follow-up for `02096e06d0`	2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin	46450995ae	app/vmselect/vmui: `make vmui-update` after `eae6063450`	2022-10-26 02:51:09 +03:00
Yury Molodov	1847a207f8	fix: change step setting field (#3270 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-26 02:47:41 +03:00
Yury Molodov	346d49ae61	vmui: limit the number of decimal places to 10 characters (#3258 ) * fix: limit the number of decimal places to 10 characters * fix: add float numbers stabilizer	2022-10-26 02:43:43 +03:00
Yury Molodov	ab880afa50	vmui: add responsive styles for small screens (#3256 ) * fix: add responsive styles for small screens * fix: correct additional settings margins * docs/CHANGELOG.md: add responsive styles	2022-10-26 02:40:41 +03:00
Aliaksandr Valialkin	c92d1fdf8e	docs/Cluster-VictoriaMetrics.md: update docs about env vars usage in command-line flags This is a follow-up for `02096e06d0`	2022-10-26 01:55:25 +03:00
Aliaksandr Valialkin	36b92f07f7	lib/envflag: allow referring environment variables in command-line flags	2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin	75990de795	docs: run `make docs-sync` after `2ed3d49c26`	2022-10-26 01:14:54 +03:00
Aliaksandr Valialkin	ecb71a7221	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin	9d0652fc6e	docs/Makefile: fix `docs-up` Makefile command after `9ccd22c1f6`	2022-10-25 17:54:14 +03:00
Denys Holius	8c0f93aa55	Docs: add guide "How to delete and replace metrics" (#2829 ) docs: add guide how to delete and replace metrics	2022-10-25 17:54:13 +03:00
Aliaksandr Valialkin	4f53147ed4	app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop Previously netstorage.MustStop() call didn't free up all the resources, so the subsequent call to nestorage.Init() would panic. This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.	2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin	596251eb87	app/vmselect/vmui: `make vmui-update` after `274e235bf7`	2022-10-24 21:29:57 +03:00
Aliaksandr Valialkin	608c977978	docs/guides/migrate-from-influx.md: properly display images at https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/guides/migrate-from-influx.md This is a follow-up for `42375679db`	2022-10-24 21:29:55 +03:00
Nguyen Van Duc	f7a1001b55	Fix images not display on key concepts document (#3266 )	2022-10-24 21:23:38 +03:00
Aliaksandr Valialkin	46a37fbeac	docs/CHANGELOG.md: document `274e235bf7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240	2022-10-24 21:03:03 +03:00
Yury Molodov	96a579c13b	vmui: optimize memory (#3255 ) * fix: change series limit logic * fix: remove spread operators	2022-10-24 21:03:02 +03:00
Yury Molodov	5de090ab43	vmui: extend options for app mode (#3252 ) * feat: add vmui customization for dbaas * feat: extends vmui customization for dbaas * fix: move input tenandId after query * fix: change serverURL when changing tenantID * fix: remove options * docs: add options description	2022-10-24 20:49:46 +03:00
Aliaksandr Valialkin	8ea84432ef	docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html	2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin	754208b124	docs/CHANGELOG.md: typo fixes	2022-10-24 17:00:18 +03:00
Aliaksandr Valialkin	a6d4711ac6	lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289	2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin	22a5405cfa	docs/CHANGELOG.md: document 69b27275d2b2bf1bdae0d8b887b44bde2a787649	2022-10-24 16:08:35 +03:00
Aliaksandr Valialkin	8946371466	vendor: `make vendor-update`	2022-10-24 15:50:57 +03:00
Aliaksandr Valialkin	51f2e473f5	lib/storage: skip blocks outside the configured retention during search Blocks outside the configured retention are eventually deleted during background merge. But such blocks may reside in the storage for long time until background merge. Previously VictoriaMetrics could spend additional CPU time on processing such blocks during search queries. Now these blocks are skipped.	2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin	2fc82b846e	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg This makes code easier to read. This is a follow-up after `d2d30581a0`	2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin	d51f9b9284	lib/storage: small code cleanups	2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin	5ace1587e6	lib/storage: re-use newTestStorage() instead of manually initializing Storage mock This is a follow-up for `d2d30581a0`	2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin	57ea7a3ee8	lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback This improves code readability a bit.	2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin	63419d8e7c	lib/storage: small refactoring: move retentionDeadline to blockStreamMerger This allows defining per-block retention in the future by updating the getRetentionDeadline function	2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin	31071347ca	lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop	2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin	5d0a91afd5	lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms	2022-10-23 12:48:43 +03:00
Aliaksandr Valialkin	2dd93449d8	lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function The searchTSIDs function was searching for metricIDs matching the the given tag filters and then was locating the corresponding TSID entries for the found metricIDs. The TSID entries aren't needed when searching for time series names (aka MetricName), so this commit removes the uneeded TSID search from the implementation of /api/v1/series API. This improves perfromance of /api/v1/series calls. This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls, since now these calls cache small metricIDs instead of big TSID entries in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs) without the need to compress the saved entries in order to save cache space. This commit also removes concurrency limiter during searching for matching time series, which was introduced in `8f16388428`, since the concurrency for all the read queries is already limited with -search.maxConcurrentRequests command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin	c92aef39b5	app/vmselect/promql: expose missing metric `vm_cache_size_max_bytes{type="promql/rollupResult"}`	2022-10-23 12:14:07 +03:00
Aliaksandr Valialkin	fe5611d6e1	lib/storage: free up memory occupied by Storage.pendingHourEntries after a temporary spike in its memory usage This reduces vmstorage memory usage by up to 20% in production workload	2022-10-21 14:59:14 +03:00
Aliaksandr Valialkin	32b6ce691b	lib/storage: move common code to newRawRowsBlock() function	2022-10-21 14:46:06 +03:00
Aliaksandr Valialkin	2f8861ed9c	lib/storage: simplify code a bit after `3f5959c053`	2022-10-21 14:39:44 +03:00
Aliaksandr Valialkin	1fb2be0cae	lib/{mergeset,storage}: simplify the code a bit after `ae55ad8749`	2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin	af648279ce	lib/storage: validate timestamps in the block only if they use encoding, which needs validation This reduces CPU usage when there is no sense in validating timestamps. This is a follow-up for `5fa9525498` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011	2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin	edf3b7be47	lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate This should improve background merge rate under high load a bit	2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin	4d71023eb9	lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes. This This change shouldn't increase memory usage for caches, since the prev cache cleaner should free up memory by deleting unused prev cache as soon as possible. See `08ca45d238` for details on prev cache cleaner.	2022-10-20 21:59:08 +03:00
Aliaksandr Valialkin	9a52b56b89	lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine This makes the code more clear after `d906d8573e`	2022-10-20 21:59:02 +03:00
Aliaksandr Valialkin	324e119172	lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals if they hang for some reason in graceful shutdown procedure.	2022-10-20 21:58:49 +03:00
Aliaksandr Valialkin	6855de311c	lib/{mergeset,storage}: avoid `unaligned 64-bit atomic operation` panic on 32-bit platforms The panic has been introduced in `68f3a02589` While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs This should improve scalability on systems with many CPU cores	2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin	526bc8a8b0	lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache This means that the majority of requests are successfully served from the current cache, so the previous cache can be reset in order to free up memory.	2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin	42cda38dbc	lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses This should improve cache scalability on systems with many CPU cores.	2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin	f22bea242f	lib/workingsetcache: randomize interval for swapping curr and prev caches This should make CPU usage smoother over time, since different caches will be swapped at different times.	2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin	0126ea31e7	docs/CHANGELOG.md: move the BUGFIX line for `1059c4d84a` into correct place	2022-10-18 20:40:37 +03:00
Nikolay	ea0596d9d8	lib/promscrape/discovery/kubernetes: correctly wrap error (#3250 ) * lib/promscrape/discovery/kubernetes: correctly wrap error follow-up after `1304824201` * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-18 20:40:37 +03:00
Roman Khavronenko	f7d69c1735	vmalert: lower severity level for RW retries (#3237 ) The message about dropped data still remains at `error` level. The change supposed to make log message more clear about how serious it is. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin	a408223e6d	vendor: `make vendor-update`	2022-10-18 10:57:06 +03:00
Aliaksandr Valialkin	574b9f4f78	app/vmselect/vmui: `make vmui-update` after `080562030d`	2022-10-18 10:51:05 +03:00
Yury Molodov	14c7223dc4	fix: remove rounding of axis limits (#3238 )	2022-10-18 10:51:03 +03:00
Aliaksandr Valialkin	d0288ea417	all: log error when environment variables referred from `-promscrape.config` are missing This should prevent from using incorrect config files	2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin	e4e2d1fcde	lib/protoparser/clusternative: allocate unmarshalWork after reading the data from input connection This shortens the time when unmarshalWork is in use. This also reduces the number of unmarshalWork objects in the pool, and its memory usage.	2022-10-18 00:24:04 +03:00
Aliaksandr Valialkin	481ca746ba	lib/protoparser/clusternative: reuse unmarshalWork in order to reduce memory allocations	2022-10-18 00:06:56 +03:00
Aliaksandr Valialkin	6f69a88a5a	lib/storage: double the number of rawRows shards on multi-core systems This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage	2022-10-17 18:19:28 +03:00
Aliaksandr Valialkin	68f3a02589	lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list	2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin	c4a3d8b169	lib/promrelabel: add relabeling tests when the source label is missing	2022-10-17 14:48:29 +03:00
Aliaksandr Valialkin	00e4c7f265	vendor: `make vendor-update`	2022-10-14 15:20:48 +03:00
Aliaksandr Valialkin	704794f164	docs/MetricsQL.md: document that input histograms must have the same set of buckets when calculating the quantile over them Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3231	2022-10-14 14:00:26 +03:00
Aliaksandr Valialkin	e3b9815fd9	docs/CHANGELOG.md: add a note that it is recommended to use v1.82.1 instead of v1.82.0	2022-10-14 13:41:29 +03:00
Aliaksandr Valialkin	7e8c9358d1	docs/CHANGELOG.md: release v1.82.1	2022-10-14 11:34:01 +03:00
Aliaksandr Valialkin	ed324aad66	lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map This prevents from possible corruption of the internal cache map when the underlying byte slice used by the string key is modified. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227	2022-10-14 09:52:18 +03:00
Denys Holius	99ec04d6aa	k8s-monitoring-via-vm-cluster.md: remove extra spaces (#3234 )	2022-10-14 09:52:16 +03:00
Yury Molodov	a50f2e141c	vmui: limit number of plotted series (#3229 ) * feat: add maximum display series by tabs * feat: add warning on PredefinedPanels.tsx * docs/CHANGELOG.md: vmui limit number of plotted series * docs/CHANGELOG.md: vmui limit number of plotted series * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-13 12:14:53 +03:00
Nikolay	07140e0877	lib/backup: set s3 default region to us-west-2 (#3224 ) * lib/backup: set s3 default region to us-west-2 it should fix an error with region detection for bucket, if AWS_REGION env var is not set * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin	c7360e1fa0	app/vmagent/remotewrite: typo fix after `c914e4dace`	2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin	9bf8d64605	app/vmselect/promql: follow-up for `930f1ee153` Document the change at docs/CHANGELOG.md Apply it to histogram_quantile() in the same way as to histogram_share() Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225	2022-10-13 12:06:24 +03:00
Siqi Liu	c0fc716426	BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225 ) Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>	2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin	8f0e4901fe	docs/CHANGELOG.md: document `b856581ad3`	2022-10-13 10:36:13 +03:00
Dan Fredell	2152d3eb61	Multi retention standardization of docs (#3223 ) * Multi retention standardization of docs While reading through the docs the implementation details had different formatting for storageNode. This standardizes them and adds a link to the retention docs. * Update docs/guides/guide-vmcluster-multiple-retention-setup.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-10-13 10:28:39 +03:00
Aliaksandr Valialkin	207b4a6c58	app/vmagent/remotewrite: typo fix after `50f5eae0e0`	2022-10-13 10:19:46 +03:00
Roman Khavronenko	895cb3e7c6	vmalert: update troubleshooting docs (#3228 ) The default value of `-datasource.queryStep` has changed, so we update the troubleshooting docs accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-13 10:14:40 +03:00
Aliaksandr Valialkin	7a6e5f9224	lib/mergeset: mention in the error message the path to the part, which triggered the error This should improve debuggability	2022-10-12 09:54:42 +03:00
Zakhar Bessarab	4e1c12a10d	doc: describe usage of env variables for obtaining credentials (#3219 )	2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin	3a14d00989	docs/CHANGELOG.md: document `a4975ace86` The original commit, which led to the issue - `877940a131`	2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin	a0f3247b14	app/vmselect/promql: properly handle zero and negative values for `-search.maxMemoryPerQuery` This is a follow-up for `04a05f161c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203	2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin	4b7ca0597d	docs/CHANGELOG.md: document 9544b5cacf7446203fac19eb7c575779dc9b280e	2022-10-12 09:33:16 +03:00
Roman Khavronenko	7fc812d3c4	vmalert: revert unexpected fileds rename during refactoring (#3222 ) Due to auto-refactoring, the filed `state` was automatically renamed to `ruleState` when the entity with the same name was renamed in other file. Reverting the change. https://github.com/VictoriaMetrics/helm-charts/issues/391 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin	54e5806d54	vendor: `make vendor-update`	2022-10-10 22:04:09 +03:00
Aliaksandr Valialkin	d6adc977b8	docs/vmbackup.md: run `make docs-sync` after `076e721c22`	2022-10-10 21:58:06 +03:00
Aliaksandr Valialkin	098c9bda27	app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203	2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin	d6a65b1f44	docs/CHANGELOG.md: document `e384d88abf`	2022-10-10 21:54:18 +03:00
Howie	d9abdc57d4	fix issue#3053 (#3182 ) vmalert: prevent duplicating label `alertname` for notifications The issue has no impact on alerting procedure. But still needs to be fixed for clarity. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3053 Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin	2ac354350c	docs/sd_configs.md: document __scrape_timeout__, __scrape_interval__ and __series_limit__ labels	2022-10-09 15:05:55 +03:00
Aliaksandr Valialkin	087393bcef	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin	d8057fca0b	docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag This is a follow-up for `5138eaeea0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203	2022-10-09 13:58:18 +03:00
Aliaksandr Valialkin	62e9dfeee6	docs/vmbackupmanager.md: update docs after adding the support to make backups to Azure blob storage This is a follow-up for `262ce77e2d`	2022-10-08 10:30:58 +03:00
Aliaksandr Valialkin	70fa9ac664	docs: document -search.maxMemoryPerQuery command-line flag	2022-10-08 01:17:49 +03:00
Aliaksandr Valialkin	938ff7bba6	app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203	2022-10-08 01:14:01 +03:00
Aliaksandr Valialkin	708d45c8fe	docs/CHANGELOG.md: add a link to a feature request for the feature, which allows specifying full scrape urls in `targets` list and in the `__address__` label Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3208	2022-10-07 23:46:34 +03:00
Aliaksandr Valialkin	3b828535f0	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin	b710354067	docs/CHANGELOG.md: document the `27ed4b853e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169 See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205	2022-10-07 23:08:23 +03:00
Yury Molodov	9e4e5abac6	vmui: auto-update chart after query field removed (#3210 ) * feat: run query after query field removed * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-07 23:08:22 +03:00
Aliaksandr Valialkin	98a4ab796c	all: update the minimum required Go verson from 1.19.1 to 1.19.2 This is needed because of security vulnerabilities found in Go 1.19.1 See https://go.dev/doc/devel/release#go1.19.2	2022-10-07 22:46:44 +03:00
Aliaksandr Valialkin	3987b0abd1	lib/promscrape: allow specifying full target url in `__address__` label Previously the `__address__` label could contain only `host:port` part of the target url, while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__` labels. Now it is possible to set the full url in `__address__` label. This makes valid the following scrape config, which is frequently used by novice users: scrape_configs: - job_name: foo static_configs: - targets: - http://host1/metrics1 - https://host2/metrics2	2022-10-07 22:46:29 +03:00
Aliaksandr Valialkin	98bd843f3c	app/{vminsert,vmselect,vmstorage}: add `package-{vminsert,vmselect,vmstorage}-{goarch}` Makefile rules for building arch-specific docker images	2022-10-07 20:17:01 +03:00
Aliaksandr Valialkin	58ee1b9edb	app/vmselect/vmui: `make vmui-update` after `a54987f671`	2022-10-07 03:31:00 +03:00
Aliaksandr Valialkin	8f28402cb3	docs/CHANGELOG.md: cut v1.82.0	2022-10-07 03:15:42 +03:00
Aliaksandr Valialkin	b736f60742	docs/CHANGELOG.md: typo fix	2022-10-07 03:11:03 +03:00
Aliaksandr Valialkin	916f1ab86c	app/vmstorage/servers: properly copy MetricName into MetricBlock inside blockIterator.NextBlock This should fix the issue at `cf36bfa189`	2022-10-07 02:57:54 +03:00
Aliaksandr Valialkin	1b2aeb621a	docs/CHANGELOG.md: add a changelog for v1.79.4 LTS release (copied from lts-1.79 branch)	2022-10-07 02:51:42 +03:00
Aliaksandr Valialkin	de49f14dce	go.mod: `go mod tidy`	2022-10-07 01:24:18 +03:00
Aliaksandr Valialkin	f926db1de4	lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511	2022-10-07 01:04:37 +03:00
Zakhar Bessarab	a5861407cc	app/vmbackup: fix compatibility with latest azure sdk (#461 )	2022-10-07 01:04:37 +03:00
Aliaksandr Valialkin	ac09a85a8b	vendor: `make vendor-update`	2022-10-07 01:04:37 +03:00
Aliaksandr Valialkin	b631b2bd64	docs/MetricsQL.md: formatting improvements Also put a link to the function type in docs for every function. This should simplify understanding of MetricsQL functions for novice users.	2022-10-07 00:54:41 +03:00
Aliaksandr Valialkin	b05ff477c8	app/vmselect/promql: properly calculate `vm_rows_scanned_per_query` histogram for rollup functions, which take into account only a few samples on the provided lookbehind window	2022-10-06 23:23:17 +03:00
Aliaksandr Valialkin	baa11a778d	app/vmselect/promql: properly calculate `quantiles_over_time()` over a single raw sample	2022-10-06 22:37:38 +03:00
Aliaksandr Valialkin	8f1e218949	docs/CHANGELOG.md: remove duplicate description of the bugfix, which has been included in v1.81.2	2022-10-06 15:53:30 +03:00
Aliaksandr Valialkin	e7c2d057e6	Makefile: remove docs/*.tmp files after running `sed` command there	2022-10-06 15:25:27 +03:00
Aliaksandr Valialkin	9f1632b30b	Makefile: add missing "=" char between "-i" flag and its value for sed This is a follow-up after `78af27f955`	2022-10-06 15:10:52 +03:00
Roman Khavronenko	92f4d6a338	docs: when modifying docs in place allow storing backups (#3205 ) The stored backups would help to identify docs corruption but aren't needed for commiting. So `.tmp` backup files are also git-ignored. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-06 15:10:52 +03:00
Roman Khavronenko	de92a8375c	vmalert: fix misleading line regarding multitenancy (#3206 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-06 15:10:52 +03:00
Aliaksandr Valialkin	371f10ec78	docs/CHANGELOG.md: add missing link to /api/v1/export docs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161	2022-10-06 15:02:39 +03:00
Aliaksandr Valialkin	958c1f291c	app: follow-up after `ec04fcac93` * Optimize fast path for /api/v1/import when importing numeric values * Move the docs about the change from features to bugfixes at docs/CHANGELOG.md * Update tests at lib/protoparser/vmimport Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161	2022-10-06 14:54:15 +03:00
Dmytro Kozlov	4064db27a8	Properly parse json when export import metric (#3180 ) * app/vmselect: properly work when export import json from `api/v1/{export, import}` API * app/vmselect: update convert function * app/vmselect: export null if `math.IsNaN(v)` * app/vmselect: get float from json * lib/protoparser: add test * docs: add change log * lib/protoparser: make export import api compatible	2022-10-06 14:54:14 +03:00
Zakhar Bessarab	db791a254b	lib/backup/s3remote: fix error checking for alternative S3 providers (#3191 )	2022-10-06 13:37:23 +03:00
Aliaksandr Valialkin	f7af86d48e	docs/Single-server-VictoriaMetrics.md: update docs after the `a54987f671`	2022-10-06 13:30:47 +03:00
Aliaksandr Valialkin	cc0d70c3d6	lib/backup/azremote: remove unused methods after the `262ce77e2d`	2022-10-06 13:30:47 +03:00
Yury Molodov	112ae2978d	vmui: maximum queries (#3196 ) vmui: allow using up 4 queries at the same time The change also introduces UI updates to make using multiple queries more conveniently.	2022-10-06 13:30:47 +03:00
Aliaksandr Valialkin	19ea0eead2	deployment/docker: update Go builder from v1.19.1 to v1.19.2 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.2+label%3ACherryPickApproved	2022-10-06 02:02:30 +03:00
Aliaksandr Valialkin	e6d866b37c	docs: follow-up after `262ce77e2d` * Document the addition of Azure blob storage support in vmbackup / vmrestore * List the supported storage system types at docs/vmrestore.md * Mention about azblob storage system support at -src and -dst command-line flags for vmbackup / vmrestore tools. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029	2022-10-06 00:36:20 +03:00
Zakhar Bessarab	6a6dcc059b	lib/backup: add support of Azure Blob Storage (#460 ) * lib/backup: add support of Azure Blob Storage * lib/backup: add enterprise support of Azure Blob Storage	2022-10-06 00:36:19 +03:00
Aliaksandr Valialkin	3fc0da2c82	docs/CHANGELOG.md: document f8ac55d70ada9ef8490b322abefb05f28f75e2e9	2022-10-06 00:06:07 +03:00
Aliaksandr Valialkin	9b1443bde5	app/vmalert: follow-up after f8ac55d70ada9ef8490b322abefb05f28f75e2e9 * Use vm_account_id and vm_project_id labels to be consistent with https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels * Document the feature that vmalert now exposes vm_account_id and vm_project_id labels if -clusterMode is set. * Use literal strings instead of string constants for vm_account_id and vm_project_id. This improves code readability.	2022-10-06 00:06:06 +03:00
Aliaksandr Valialkin	1459201de5	app/vmbackupmanager: expose `vm_backup_in_flight` metrics (follow-up after c6bca4a0b47b4f5626e1913d3480e62d657ed4cf)	2022-10-05 23:30:39 +03:00
Aliaksandr Valialkin	98d58fdb57	app/vmalert: update -external.alert.source command-line flag description after `61544e13ad`	2022-10-05 22:54:23 +03:00
Roman Khavronenko	6f6f6afae0	vmalert: allow using `{{$labels}}` for templating in `-external.alert.source` (#3194 ) The change is supposed to provide additional flexibility for generating alert's source link based on label values. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-05 22:53:02 +03:00
Zakhar Bessarab	434b00cee8	docker-compose: move `TooManyLogs` into `vm-health` alerts set (#3199 )	2022-10-05 22:42:31 +03:00
Dmytro Kozlov	8469670fb2	docs: add cardinality explorer blog post (#3198 ) docs: add cardinality explorer blog post	2022-10-05 22:41:36 +03:00
Aliaksandr Valialkin	8b63f84a86	Makefile: fix sed command and if condition for Linux bash after `2d11896486`	2022-10-05 22:40:43 +03:00
Roman Khavronenko	238401e3ed	docs: udpate Datadog section (#3190 ) The `docs-sync` command was updated to modify images path for assets in `docs/` folder. The change allows to refer images from `docs` without copying them to the root folder. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-10-05 22:36:35 +03:00
Yurii Kravets	39b85119f2	docs: Update How to send data from DataDog agent (#3168 ) docs: Update How to send data from DataDog agent	2022-10-05 22:24:42 +03:00
Aliaksandr Valialkin	43bdd96a6e	app/vmselect: improve performance scalability on multi-CPU systems for `/api/v1/export/...` endpoints	2022-10-01 22:16:07 +03:00
Aliaksandr Valialkin	5497997b72	app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores Use sync.Map instead of a global mutex there. This should lift scalability limits on systems with many CPU cores.	2022-10-01 20:14:14 +03:00
Aliaksandr Valialkin	972fcfcd17	app/vmselect/prometheus: improve scalability of `/federate` endpoint on systems with many CPU cores Minimize usage of global lock inside bufferedwriter.Write() when processing `/federate` data on systems with many CPU cores	2022-10-01 20:14:13 +03:00
Aliaksandr Valialkin	ae6c7edf04	app/vmselect: do not export NaN values for stale metrics at /federate endpoint Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185	2022-10-01 19:48:21 +03:00
Aliaksandr Valialkin	e6ee725bae	docs/Cluster-VictoriaMetrics.md: add a security note for multitenant support via labels	2022-10-01 18:56:56 +03:00
Aliaksandr Valialkin	e9f1213cc9	docs/vmagent.md: fix incorrect url for multitenant writes to VictoriaMetrics cluster	2022-10-01 18:51:57 +03:00
Aliaksandr Valialkin	b857365b84	app/vmagent/remotewrite: allow specifying per-`-remoteWrite.url` disk limits for persistent queue with pending data This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071 Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970	2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin	6f9ce3f6d6	lib/flagutil: rename Array to ArrayString This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:28:19 +03:00
Aliaksandr Valialkin	b57ca71eb2	vendor: `make vendor-update`	2022-10-01 17:19:01 +03:00
Zakhar Bessarab	5b7e8d1309	vmbackup: update AWS SDK to v2 (#3174 ) * lib/backup/s3remote: update AWS SDK to v2 * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * lib/backup/s3remote: refactor error handling Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-01 17:13:04 +03:00
Aliaksandr Valialkin	93e84a1c57	lib/httpserver: use 302 redirects instead of 301 redirects Incorrect 301 redirects can be cached by user agents such as web browsers. This can complicate recovery procedure after the incorrect redirect is fixed, e.g. web browser cache must be reset. The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752	2022-10-01 16:56:43 +03:00
Aliaksandr Valialkin	e03a924236	app/vmauth: do not remove trailing slash from the proxied path This should fix the issue with opening VMUI at /vmui/ page. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752	2022-10-01 16:56:42 +03:00
Aliaksandr Valialkin	f0a748a3aa	lib/promscrape/discovery/azure: remove unneeded conversion to string	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	735de9ee54	lib/promscrape: add `external_labels` from `global` section of `-promscrape.config` after the relabeling is applied to the scraped metrics This aligns with Prometheus behaviour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	e5aa34b2e3	lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	d81285c814	app/vmagent: accept requests with `/prometheus` and `/influx` path prefixes in the same way as VictoriaMetrics does This allows using vmagent as a drop-in replacement for VictoriaMetrics for push protocols.	2022-10-01 14:11:09 +03:00
Aliaksandr Valialkin	b5fe42514d	docs/vmagent.md: formatting fixes	2022-10-01 13:36:48 +03:00
Aliaksandr Valialkin	9676519830	app/vmagent/README.md: apply typo fix from `1fda517af9`	2022-10-01 12:31:42 +03:00
lionelee	f9843442f2	docs/vmagent.md: fix typo (#3186 )	2022-10-01 12:31:41 +03:00
Aliaksandr Valialkin	b96fe2e265	lib/storage: optimize matching speed for non-trivial regexp filters Wrap re.Match into bytesutil.FastStringMatcher. This increases performance for `{foo=~"complex_regex_here"}` filters by up to 4x.	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	72d8653233	docs/CHANGELOG.md: clarify the description of the improvement in relabeling performance	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	969ae90941	lib/promrelabel: remove redundant memory allocations by using interned strings	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	d8d455856c	lib/promrelabel: add a benchmark for realistic Kubernetes relabeling The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes This benchmark has been copied from `d521933053/model/relabel/relabel_test.go (L505)` See also https://github.com/prometheus/prometheus/pull/11147	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	c628f5b6eb	lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does See https://github.com/prometheus/prometheus/pull/11326	2022-09-30 20:49:08 +03:00
Aliaksandr Valialkin	ae2f669e73	Makefile: run errcheck for all the app/... subdirs	2022-09-30 18:40:06 +03:00
Roman Khavronenko	408d7043a1	vmalert: support auth configs per static_target (#3188 ) Allow configuring authorization params per list of targets in vmalert's notifier config for `static_configs`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-30 18:38:11 +03:00
Aliaksandr Valialkin	c8000c029a	app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints These labels are substituted by documented vm_account_id and vm_project_id labels. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels This is a follow up for `505d359b39`	2022-09-30 18:06:00 +03:00
Nikolay	505d359b39	app/vminsert: allows parsing tenant id from labels (#3009 ) * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin	39ba55dbb3	lib/promrelabel: go fmt	2022-09-30 12:28:27 +03:00
Aliaksandr Valialkin	9fc2817f41	lib/promrelabel: optimize `action: replace` for non-trivial regex values Cache `action: replace` results for non-trivial regexs and return them next time instead of performing CPU-intensive regex replacement. Optimize also `action: labelmap_all` and `action: replace_all` in the same way.	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	f38c9db74d	lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at `17289ff481`	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	fa46c28c5f	lib/promrelabel: optimize `action: labelmap` for non-trivial regexs	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	b4bb1477fe	lib/regexutil: cache MatchString results for unoptimized regexps This increases relabeling performance by 3x for unoptimized regexs	2022-09-30 12:28:25 +03:00
Dmytro Kozlov	d0a9fff70c	docs, app/vmgateway: add description about new `auth.httpHeader` flag (#3134 ) * docs, app/vmgateway: add description about new `auth.httpHeader` flag * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 08:56:43 +03:00
Aliaksandr Valialkin	98175493d7	app/vmselect/promql: remove empty series before applying aggregate function Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions. Such series must be ingored by all the aggregate functions. So it is better from consistency PoV filtering out empty series before applying aggregate functions.	2022-09-30 08:44:06 +03:00
Roman Khavronenko	273472dc20	app/vmselect: ignore empty series for `limit_offset` (#3178 ) * app/vmselect: ignore empty series for `limit_offset` VictoriaMetrics doesn't return empty series (with all NaN values) to the user. But such series are filtered after transform functions. It means `limit_offset` will account for empty series as well. For example, let's consider following data set: ``` time series: foo{label="1"} NaN, NaN, NaN, NaN // empty series foo{label="2"} 1, 2, 3, 4 foo{label="3"} 4, 3, 2, 1 ``` When user requests all series for metric `foo` the empty series will be filtered out: ``` /query=foo: foo{label="v2"} 1, 2, 3, 4 foo{label="v3"} 4, 3, 2, 1 ``` But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet. So it will return `foo{label="v2"}` (skips the first in list) ``` /query=limit_offset(1, 1, foo): foo{label="v2"} 1, 2, 3, 4 ``` Expected result would be to apply `limit_offset` to already filtered list, so in result we receive `foo{label="v3"}`: ``` /query=limit_offset(1, 1, foo): foo{label="v3"} 4, 3, 2, 1 ``` The change does exactly that - filters empty series before applying `limit_offset`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmselect: ignore empty series for `limit_offset` Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-30 08:44:05 +03:00
Aliaksandr Valialkin	f1eebc0a99	lib/promrelabel: properly parse regex with escaped $ at the end Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131 Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179	2022-09-30 08:20:57 +03:00
Aliaksandr Valialkin	33590c5fb9	docs/CHANGELOG.md: document `39f559d22b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013	2022-09-30 07:49:54 +03:00
Nikolay	909709346e	lib/awsapi: fixes sign encoding (#3183 ) * lib/awsapi: fixes sign encoding previously white spaces at filter were incorrectly encoded encoding tip was copied from aws signing lib For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171 * Update lib/awsapi/sign.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 07:49:18 +03:00
Roman Khavronenko	a2ded58600	vmalert: allow using extra labels in annotations (#3181 ) According to Ruler specification, only labels returned within time series should be available for use in annotations. For long time, vmalert didn't respect this rule. And in PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2403 this was fixed for the sake of compatibility. However, this resulted into users confusion, as they expected all configured and extra labels to be available - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013 This fix allows to use extra labels in Annotations. But in the case of conflicts the original labels (extracted from time series) are preferred. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-30 07:48:59 +03:00
Aliaksandr Valialkin	c0aa10bd73	lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil lib/bytesutil is more appropriate place for InternString() function	2022-09-30 07:34:59 +03:00
nemobis	6723bdf867	docs: fix typo in Cluster-VictoriaMetrics.md (#3172 )	2022-09-28 10:43:17 +03:00
panguicai	156e7035c7	docs: fix typo for vmalert docs (#3173 ) Signed-off-by: panguicai008 <1121906548@qq.com>	2022-09-28 10:42:13 +03:00
Roman Khavronenko	3f78608da2	docs: update img for key concepts (#3176 ) Make dots bigger for range_query, so it will be easier to read the graph. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-28 10:41:35 +03:00
Aliaksandr Valialkin	4afa25fb38	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	9c6c691471	lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings This is a follow-up for `7c2474dac7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	7f0b95b50a	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for `fcffdba9dc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin	41882222d3	lib/netutil/tls.go: consistently use tlsMinVersion name across source code This should simplify further code maintenance and refactoring This is a follow-up after `6ab1cede62`	2022-09-26 17:59:07 +03:00
Dmytro Kozlov	28dcff5791	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109 ) * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:38:43 +03:00
Denys Holius	3f38a88f71	docs: managed_victoriametrics/quickstart.md fix image link (#3165 )	2022-09-26 17:38:41 +03:00
Roman Khavronenko	b8c4c36b89	docs: clarify numeric label values for `label_value` (#3129 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3114 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 16:49:45 +03:00
Max Golionko	322e5006d2	added security policy (#3140 ) * added security policy * Update SECURITY.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:47:38 +03:00
Denys Holius	ebfccada32	Adds packer build for server with VM Single node in vultr.com marketplace (#3142 ) * adds packer build for server with VM Single node in vultr.com marketplace * fix missed varibale	2022-09-26 16:45:29 +03:00
Roman Khavronenko	fe71c73fe1	lib/mergeset: follow-up after `a0e7432e42` (#3145 ) * lib/mergeset: follow-up after `a0e7432e42` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:43:17 +03:00
Aliaksandr Valialkin	d91bf89651	docs/CHANGELOG.md: document `f022296d96` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147	2022-09-26 16:32:54 +03:00
Zakhar Bessarab	6c65ee18d9	vmbackup: configure retries for GCS remote FS (#3156 )	2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin	6611417732	vendor: `make vendor-update`	2022-09-26 15:47:35 +03:00
Aliaksandr Valialkin	89e1060101	docs/CHANGELOG.md: document `166d444159`	2022-09-26 15:39:51 +03:00
Roman Khavronenko	4acbf6e7d4	vmselect/rollup: rm workaround for slow-changing counters (#3163 ) The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962. However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function. Removing the workaround makes VM to produce accurate results when calculating `delta` or `increase` functions over slow-changing counters with vary intervals between data points. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 15:39:49 +03:00
Aliaksandr Valialkin	2b98f2bc1a	lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102 See the specification https://graphite.readthedocs.io/en/latest/tags.html	2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin	938d7a932c	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics sanitizes metric names in DataDog data by default This is a follow-up for `7c2474dac7`	2022-09-26 14:03:26 +03:00
Aliaksandr Valialkin	dbc20091b1	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin	6a6cf9c590	app/{vmagent,vminsert}: add `-usePromCompatibleNaming` command-line flag for normalizing metric names and label names in the ingested samples Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113 Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146	2022-09-26 13:13:31 +03:00
Aliaksandr Valialkin	594a4ab345	docs/vmalert.md: follow-up for `0c95f928ae` - Clarify the description for -datasource.queryStep command-line flag - Consistently use a single dash in front of -datasource.queryStep command-line flag - Update -help output at docs/vmalert.md	2022-09-26 08:49:47 +03:00
Aliaksandr Valialkin	a6a869c365	docs/vmalert.md: follow-up after `7748a9d629` - Consistently use single dash in front of command-line flags instead of double dashes. - Add a warning that too small -search.latencyOffset may lead to incomplete query results.	2022-09-26 08:49:47 +03:00
Aliaksandr Valialkin	2fe38a3243	docs: follow-up for `03d54ac890` - Typo fix: 'adn' -> 'and' - Remove duplicate link to https://victoriametrics.com/blog/victoriametrics-monitoring/ from docs/Articles.md . The initial link has been already added in `27254096b2`	2022-09-26 08:49:47 +03:00
Roman Khavronenko	0efc20d7b8	dashboards: replace `Index size` panel with `Active series` (#3157 ) Panel `Index size` showed itself impractical for users. So replacing it with `Active series` panel. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/776#issuecomment-1255823734 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 08:48:25 +03:00
Roman Khavronenko	cffceba0f5	vmalert: set default value for `datasource.queryStep` to `5m` (#3149 ) Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`. Param `step` is added by vmalert to every rule evaluation request sent to datasource. Before this change, `step` was equal to group's evaluation interval by default. Param `step` for instant queries defines how far VM can look back for the last written data point. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 08:40:17 +03:00
Roman Khavronenko	b86cf7d707	vmalert: add info about `search.latencyOffset` to Troubleshooting (#3151 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 08:38:23 +03:00
Roman Khavronenko	7ba51c57ae	deployment/docker: fix image versions for cluster components (#3150 ) Cluster components always have `-cluster` suffix. The change fixes incorrect image tag in docker-compose manifest. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 08:29:58 +03:00
Roman Khavronenko	c027991caf	docs: mention VictoriaMetrics Monitoring blog post (#3152 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-26 08:29:02 +03:00
Aliaksandr Valialkin	722c5a4625	docs/Articles.md: add a link to https://victoriametrics.com/blog/victoriametrics-monitoring/	2022-09-22 19:18:51 +03:00
Aliaksandr Valialkin	8da1b4fc42	docs/Articles.md: add a link to https://dataswamp.org/~solene/2022-09-11-exploring-monitoring-stacks.html	2022-09-22 19:18:51 +03:00
Denys Holius	99777b3d2a	deployment/docker: add version tag for docker containers (#3141 ) * deployment/docker/docker-compose.yml: adds version tags for VictoriaMetrics containers * deployment/docker/docker-compose-cluster.yml: adds version tags for VictoriaMetrics containers	2022-09-22 19:18:50 +03:00
匠心零度	9777c7a367	lib/querytracer: fix comment (#3135 )	2022-09-22 13:59:17 +02:00
Aliaksandr Valialkin	101581e053	deployment/docker/provisioning/dashboards: sync with master branch after `5714a68ac6`	2022-09-21 12:06:01 +03:00
Aliaksandr Valialkin	067755a8cf	README.md: sync with docs/Cluster-VictoriaMetrics.md after `5714a68ac6`	2022-09-21 12:03:10 +03:00
Aliaksandr Valialkin	1271010727	vendor: `make vendor-update`	2022-09-21 12:03:10 +03:00
Roman Khavronenko	f772ee8326	deployment/docker: move cluster compose env to master branch (#3130 ) * deployment/docker: move cluster compose env to master branch The change supposed to simplify the process of maintaining for single/cluster docker-compose envs, alerts, dashboards. It also supposes to reduce confusion for users when looking for cluster related alerts/configs. Signed-off-by: hagen1778 <roman@victoriametrics.com> * deployment/docker: move cluster compose env to master branch Review updates. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-21 12:03:10 +03:00
Dmytro Kozlov	ed842e7d3a	app/{vmctl,vmalert}: update progress bar library (make vendor-update) (#3138 ) * app/{vmctl,vmalert}: update progress bar library (make vendor-update) * app/{vmctl,vmalert}: make vendor-update	2022-09-21 11:11:40 +03:00
Roman Khavronenko	74e81d31a7	vmalert: prodvide more details on duplicates (#3136 ) Now vmalert will print the following messages on dupliсates: ``` "recording rule \"record\"; expr: \"up == 1\"; labels: summary={{ value\|query }}" is a duplicate within the group "test" "alerting rule \"alert\"; expr: \"up == 1\"; labels: description={{ value\|query }}" is a duplicate within the group "test" ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3127 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-21 11:11:40 +03:00
Aliaksandr Valialkin	336007182c	app/vmagent/remotewrite: `go fmt` after `2b55d167d7`	2022-09-19 15:14:03 +03:00
Aliaksandr Valialkin	7437d3c48f	vendor: `make vendor-update`	2022-09-19 15:14:01 +03:00
Aliaksandr Valialkin	47ebf6e836	Makefile: fix -compat value passed to `go mod tidy`	2022-09-19 15:14:01 +03:00
Aliaksandr Valialkin	1b99501798	docs/managed_victoriametrics/quickstart.md: small fixes after `606166ef68`	2022-09-19 15:04:37 +03:00
Roman Khavronenko	5dfe63e102	Dashboards (#3120 ) * dashboards/cluster: few updates * apply consistent formatting across panels; * make resource usage panels per component more detailed; * add extra panels to vmselect for displaying `vm_rows_read_per_query`, `vm_rows_scanned_per_query`, `vm_rows_read_per_series` and `vm_series_read_per_query` metrics. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/single: few updates * apply consistent formatting across panels; * add extra panels to Performance for displaying `vm_rows_read_per_query`, `vm_rows_scanned_per_query`, `vm_rows_read_per_series` and `vm_series_read_per_query` metrics. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/vmagent: few updates * apply consistent formatting across panels; * add panels for showing number of samples ingested or scraped; * adapt resource usage panels for multiple selected jobs/instances; * add adhoc variable; * display vmagent's version in Stats. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/vmalert: few updates * apply consistent formatting across panels; * adapt resource usage panels for multiple selected jobs/instances; * show vmalert version in Stats section. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-19 15:04:37 +03:00
Aliaksandr Valialkin	19dc29abd7	docs/CHANGELOG.md: clarify the change at `622bbedbe1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119	2022-09-19 15:04:37 +03:00
Roman Khavronenko	360b022603	vmalert: always re-evaluate Annotations (#3119 ) * vmalert: always re-evaluate Annotations Previously, Annotations were evaluated only: 1. On alert creating. 2. On alert's value change. This is premature optimization. It was assumed that since annotations could contain only text with alert's labels or value - there is no need in spending resources to re-compile Annotations. Later, template function `query` was added, which can execute arbitrary queries and return different results on every evaluation. So if it was used in annotations, it would be executed only on init or value change. Another case when optimization caused an issue - annotations hot reload. In this case, annotations of the active alert won't change even if Rule's annotations were changed. This fix enables Annotations re-evaluation on each iteration to resolve issues above. It would have some impact on performance, but it is unlikely it will be noticeable. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: add tp Changelog Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-19 15:04:37 +03:00
Roman Khavronenko	1c13cce5ed	vmalert: add Troubleshooting section to docs (#3115 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-19 15:04:37 +03:00
Dmytro Kozlov	8ca42b9bcb	docs/managed_victoriametrics: add how to restore password section (#3116 ) docs/managed_victoriametrics: add how to restore password section	2022-09-19 15:04:37 +03:00
Roman Khavronenko	09e211a05f	vmalert: print example of `curl` command for rule's state (#3112 ) The change adds an example of `curl` command to the Rule's page. The command is generated for each recorded state. It is supposed user can just copy&execute the command to see what was returned to vmalert. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-19 15:04:37 +03:00
Aliaksandr Valialkin	34db3fdd3f	app/vmagent/remotewrite: add benchmarks for comparing the performance of standard Snappy encoder with github.com/klauspost/compress/s2 encoder The standard Snappy encoder from github.com/golang/snappy shows quite good performance number for compressing the Prometheus remote_write proto messages according to the added benchmarks, so there is no need in switching to github.com/klauspost/compress/s2 yet.	2022-09-19 14:27:56 +03:00
Dmytro Kozlov	52a79a901e	app/vmctl: add description about `influx-skip-database-label` flag (#3111 )	2022-09-15 13:24:16 +03:00
Aliaksandr Valialkin	27a6ac5a30	docs/Cluster-VictoriaMetrics.md: update `-help` output after explicit marking of enterprise flags	2022-09-15 13:19:11 +03:00
Aliaksandr Valialkin	713ebb8297	docs: update `-help` output after explicit mentioning of enterprise flags	2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin	d77eb5170c	docs/vmauth.md: update `-help` output after explicit marking of enterprise flags	2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin	424bcfc17b	docs/vmalert.md: update `-help` output after explicit marking of enterprise flags	2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin	1275e0580e	docs: update docs with explicitly marked enterprise command-line flags for VictoriaMetrics and vmagent	2022-09-15 13:19:01 +03:00
Aliaksandr Valialkin	6f1f49c64a	docs/FAQ.md: add an answer for `What is the difference between single-node and cluster versions of VictoriaMetrics?`	2022-09-15 09:57:12 +03:00
Dmytro Kozlov	36ea8537d9	vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982 ) * vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) * vmselect/promql: fix tests, add documentation * vmselect/promql: update test * vmselect/promql: update for alphanumeric sorting, fix tests * vmselect/promql: remove comments * vmselect/promql: cleanup * vmselect/promql: avoid memory allocations, update functions descriptions * vmselect/promql: make linter happy (remove ineffectual assigment) * vmselect/promql: add test case, fix behavior when strings are equal * vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0 this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions * wip * lib/promscrape: read response body into memory in stream parsing mode before parsing it This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage. * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-14 17:42:07 +03:00
Aliaksandr Valialkin	73714c7516	docs/Articles.md: add a link to https://www.groundcover.com/blog/prometheus-grafana-kubernetes	2022-09-14 15:11:48 +03:00
Aliaksandr Valialkin	d1b9cbcef4	lib/promscrape: typo fix after `74c00a8762`	2022-09-14 15:08:42 +03:00
Roman Khavronenko	6ae4f3526b	vmalert: add experimental feature of storing Rule's evaluation state (#3106 ) vmalert: add experimental feature of storing Rule's evaluation state The new feature keeps last 20 state changes of each Rule in memory. The state are available for view on the Rule's view page. The page can be opened by clicking on `Details` link next to Rule's name on the `/groups` page. States change suppose to help in investigating cases when Rule doesn't generate alerts or records. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-14 15:08:40 +03:00
Roman Khavronenko	d071e39694	bump Go version to 1.19.1 (#3108 ) The reason is to cover vulnerability GO-2022-0969 Found in: net/http@go1.18.5 Fixed in: net/http@go1.19.1 More info: https://pkg.go.dev/vuln/GO-2022-0969 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-14 13:43:27 +03:00
Aliaksandr Valialkin	b6bdbb4787	app/vmselect/vmui: `make vmui-update` after `1304824201`	2022-09-14 13:32:43 +03:00
Aliaksandr Valialkin	2351468bc4	lib/promscrape: read response body into memory in stream parsing mode before parsing it This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:29:39 +03:00
Denys Holius	a186f21f4c	deployment/docker/docker-compose: expose port for vminsert (#3096 ) deployment/docker: open port for vminsert because it can't be reached from nginx/curl/etc as it run at random port in OS	2022-09-14 07:13:56 +02:00
Aliaksandr Valialkin	592612b63f	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error This should improve debugging issues with Kubernetes API server	2022-09-13 19:37:40 +03:00
Yury Molodov	46e8bd34f7	vmui: fix query params saving in URL (#3104 )	2022-09-13 19:19:57 +03:00
Aliaksandr Valialkin	961e986ca9	docs: consistently use docs.victoriametrics.com instead of victoriametrics.github.io in all the links	2022-09-13 16:50:36 +03:00
Aliaksandr Valialkin	fea576e061	vendor: `make vendor-update`	2022-09-13 16:45:00 +03:00
Aliaksandr Valialkin	20834c1757	app/vmalert: follow-up after `8441375da2` - Rename logDebug() to logDebugf() and pass format string together with format args directly to logDebugf(). This eliminates fmt.Sprintf() overhead at logDebug() call site when debugging is disabled. - Format labels in debug message in Prometheus format, e.g. {label1="value1",...labelN="valueN"} Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025	2022-09-13 16:36:31 +03:00
Roman Khavronenko	a887c1bc07	vmalert: add `debug` mode for alerting rules (#3055 ) * vmalert: add `debug` mode for alerting rules Debug information includes alerts state changes and requests sent to the datasource. Debug can be enabled only on rule's level. It might be useful for debugging unexpected behaviour of alerting rule. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: review fixes Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update app/vmalert/alerting.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * vmalert: go fmt Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-13 16:36:30 +03:00
Craig Rodrigues	6b5843eb66	docs: Use headings in DataDog section to make things easier to read (#3089 ) Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org> Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>	2022-09-13 16:19:14 +03:00
Aliaksandr Valialkin	6a44e1bc06	docs/CHANGELOG.md: document atomic directory deletion Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:16:12 +03:00
Aliaksandr Valialkin	5b488a339d	lib/mergeset: atomically remove part dirs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin	fe52378f45	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic	2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin	6c9729d694	lib/storage: atomically remove parts inside partitions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:28:41 +03:00
Aliaksandr Valialkin	7d26c37239	docs/ExtendedPromQL.md: move the page to the bottom of the contents list at http://docs.victoriametrics.com	2022-09-13 15:12:15 +03:00
Aliaksandr Valialkin	28545eb630	Revert "docs/ExtendedPromQL.md: remove outdated doc" This reverts commit `971e3d83f7`. Reason for revert: the ExtendedPromQL doc is still referred from third-party sites.	2022-09-13 15:10:03 +03:00
Aliaksandr Valialkin	daa42e4f79	lib/storage: atomically remove partitions, which went outside the configured retention Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin	0a342f04b2	lib/storage: properly remove cache directory contents if `reset_cache_on_startup` file is located there Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 13:32:05 +03:00
Aliaksandr Valialkin	ff7188b6a5	lib/storage: atomically remove snapshot directories Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:25:48 +03:00
Yury Molodov	465894d720	vmui: fix data processing (#3092 ) * fix: change data processing * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-12 10:56:39 +03:00
Yury Molodov	6494808a24	fix: change columns for Top Queries (#3093 )	2022-09-12 10:46:48 +03:00
Aliaksandr Valialkin	3a5b27dc2d	docs/FAQ.md: add a link to https://victoriametrics.com/blog/mimir-benchmark/ to `VictoriaMetrics vs Mimir` chapter	2022-09-12 10:38:11 +03:00
Aliaksandr Valialkin	2c0c87ee79	docs/Articles.md: add a link to https://victoriametrics.com/blog/mimir-benchmark/	2022-09-12 10:38:10 +03:00
Yury Molodov	c8de98e03f	vmui: add lists of top queries (#3065 ) * feat: add lists of top queries * fix: change the field label * refactor: add handlers for readability * app/vmselect: `make vmui-update` * docs: document `top queries` tab Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-08 21:44:43 +03:00
John Belmonte	eb0b2ef3d9	MetricsQL doc spellcheck (#3080 )	2022-09-08 21:28:49 +03:00
Aliaksandr Valialkin	052b527b39	docs/CHANGELOG.md: document `ec273eafef` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3076	2022-09-08 21:21:26 +03:00
Dmytro Kozlov	ec273eafef	app/{vmselect, vminsert}: fails with error when user defines equal addresses in the -storageNodes flag (#3082 )	2022-09-08 21:17:58 +03:00
Aliaksandr Valialkin	21d608b210	Makefile: remove github-create-release and github-upload-assets commands from `publish-release` This is a follow-up for `b9231c715a`	2022-09-08 21:03:00 +03:00
Aliaksandr Valialkin	7ffd3ae1ff	vendor: `make vendor-update`	2022-09-08 18:57:19 +03:00
Aliaksandr Valialkin	6cbe85a08a	Makefile: consistently use `go install` instead of `go get` for installing various binaries needed during build/test/check of the code `go install` is the preferred way for installing go binaries starting from the minimum supported Go version for VictoriaMetrics - Go1.18 - see https://tip.golang.org/doc/go1.18#go-command	2022-09-08 18:46:08 +03:00
Aliaksandr Valialkin	e29c9dea30	.github/workflows/main.yml: stop setting GO111MODULE=on env var, since it is unnecessary in Go1.18 and newer versions	2022-09-08 18:46:05 +03:00
Aliaksandr Valialkin	c083286a92	Makefile: check for vulnerabilities in used Go packages with govulncheck when running `make check-all` See https://go.dev/blog/vuln	2022-09-08 18:46:00 +03:00
Aliaksandr Valialkin	e619fa8b38	deployment/docker: update Go builder for prod binaries from Go1.19.0 to Go1.19.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.1+label%3ACherryPickApproved	2022-09-08 18:27:55 +03:00
Aliaksandr Valialkin	f6b3ad0c37	docs/Release-Guide.md: require manual push of the created release tags to public Github The automated push of release tags to Github require specifying the remote repository name when doing `git push <remote-repo-name> v1.xx.y`. The remote repository name can differ in different environments, so it cannot be put into Makefile rule. TODO: create a Makefile rule, which generates standard remote names for public and private repositories in Git, so `git push` for release tags could be automated then.	2022-09-08 15:00:34 +03:00
Aliaksandr Valialkin	f0eea5b02d	app/vmselect/netstorage: fix a typo, which leads to incorrect query results in VictoriaMetrics cluster The typo has been introduced in the commit `1a254ea20c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067	2022-09-08 13:46:40 +03:00
Max Golionko	e07f23a1b9	moved cluster dashboard to master (#3074 ) dashboards: move cluster dashboard to master branch This change should simplify dashboards management.	2022-09-08 11:47:25 +03:00
Aliaksandr Valialkin	6ebe4f5392	docs/Single-server-VictoriaMetrics.md: add a note that cardinality explorer at cluster version of VictoriaMetrics may return lower than expected number of unique label values See the corresponding comment in the code: `5a6e617b5e/app/vmselect/netstorage/netstorage.go (L1039-L1045)`	2022-09-06 14:48:21 +03:00
Dmytro Kozlov	96ecec877d	vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049 ) * vmselect/{promql, prometheus}: show flag names which user can update in error message * vmselect/{promql, prometheus}: fix typo	2022-09-06 14:48:20 +03:00
Aliaksandr Valialkin	5a6e617b5e	docs/Articles.md: add https://tech.bedrockstreaming.com/2022/09/06/monitoring-at-scale-with-victoriametrics.html	2022-09-06 13:33:07 +03:00
Aliaksandr Valialkin	f6990871a3	docs/vmctl.md: `make docs-sync` after `c5261d5f56` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733	2022-09-06 13:20:16 +03:00
Zakhar Bessarab	ef01f01b6c	vmctl: implement support of chunking data for vm-native export (#3044 ) vmctl: implement support of chunking data for vm-native export process See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733	2022-09-06 13:14:48 +03:00
Craig Rodrigues	d02f09700f	docs: Clarify DataDog examples for VictoriaMetrics cluster (#3048 ) Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>	2022-09-06 13:12:25 +03:00
Yurii Kravets	b47adb3e3e	docs: Update keyConcepts (#3040 ) docs: update keyConcepts Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-09-06 13:10:26 +03:00
Aliaksandr Valialkin	051e722112	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:07:49 +03:00
Aliaksandr Valialkin	b7f3569522	app/vmselect/prometheus: follow-up after `50e2524bc2` - Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values - Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values - Document the change at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052	2022-09-05 11:57:08 +03:00
匠心零度	65d1124dc4	api prometheus/api/v1/label/../values time not specified, (#3052 ) modify default start values	2022-09-05 11:51:32 +03:00
Aliaksandr Valialkin	c6b74148cf	app/vmselect/promql: consistently calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045	2022-09-02 23:19:05 +03:00
Aliaksandr Valialkin	6fe1f895f0	Makefile: properly push public tags	2022-09-02 23:19:03 +03:00
Aliaksandr Valialkin	569843c0dd	docs/CHANGELOG.md: cut v1.81.1	2022-09-02 22:00:34 +03:00
Aliaksandr Valialkin	0be941c99b	Makefile: push v1.xx.y and v1.xx.y-cluster tags to github before creating the v1.xx.y release at github Otherwise Github creates the v1.xx.y tag on itself when creating the release	2022-09-02 21:49:10 +03:00
Aliaksandr Valialkin	6837400c5a	vendor: `make vendor-update`	2022-09-02 21:43:12 +03:00
Aliaksandr Valialkin	9cca3a0a1b	app/vmselect/netstorage: fix potential panic under high load The panic may trigger during data blocks' processing received from vmstorage nodes when some of vmstorage nodes return an error or when `-replicationFactor` is set to values higher than 2 at `vmselect`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058	2022-09-02 21:36:15 +03:00
Aliaksandr Valialkin	024e2f18da	app/vmselect/promql: evaluate `union()` args in parallel in order to increase query performance Note that the parallel execution of `union()` args may take more memory and CPU time than the sequential execution if args contain heavy queries, which may load all the available CPU, disk and memory resources and vmselect and vmstorage levels.	2022-09-02 21:01:04 +03:00
Aliaksandr Valialkin	a580d7ac13	docs/Articles.md: add a link to https://aatarasoff.medium.com/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b	2022-08-31 05:03:57 +03:00
Aliaksandr Valialkin	33b2a1df45	docs/FAQ.md: mention that single-node vm is easier to setup and operate than the cluster version of vm	2022-08-31 05:03:56 +03:00
Aliaksandr Valialkin	a11eae3a76	docs/CHANGELOG.md: cut v1.81.0	2022-08-31 02:33:36 +03:00
Max Golionko	0359fce052	simplify release process (#3012 ) * simplify release process * address comments * address comments * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-31 02:28:45 +03:00
Aliaksandr Valialkin	30a1d0c622	docs/CHANGELOG.md: add missing link to vmagent	2022-08-31 02:22:52 +03:00
Craig Rodrigues	6e057faf1b	docs: Add DataDog to list of ingestion methods (#3047 ) Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org> Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org>	2022-08-31 00:23:07 +03:00
Aliaksandr Valialkin	cdb22c5ca3	docs/CHANGELOG.md: document v1.79.3 LTS release	2022-08-31 00:15:26 +03:00
Craig Rodrigues	ee988cc71c	docs: add https://github.com/VictoriaMetrics/ansible-playbooks to README.md (#3046 ) Signed-off-by: Craig Rodrigues <rodrigc@crodrigues.org> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2022-08-31 00:13:26 +03:00
Aliaksandr Valialkin	74efe00649	docs: clarify why cluster version is recommended to use for serving high loads	2022-08-30 13:21:43 +03:00
Aliaksandr Valialkin	be5299be2c	docs/FAQ.md: add chapters explaining main differences between single-node and cluster versions of VictoriaMetrics - Which type of VictoriaMetrics is better to use in production? - How to migrate data from single-node VictoriaMetrics to cluster?	2022-08-30 13:12:57 +03:00
Artem Navoiev	1f5efd2a4c	update multi-region guide, specify multi-level vmselect option (#3039 ) * update multi-region guide, specify multi-level vmselect option Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * add more references * Apply suggestions from code review Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-30 12:34:53 +03:00
匠心零度	80067fd03a	reduce unnecessary vmstorage query (#3031 ) * reduce unnecessary vmstorage query * reduce unnecessary vmstorage query * rollback limit logic /api/v1/label/*	2022-08-30 12:34:23 +03:00
Denys Holius	6f92fb8783	04-install-victoriametrics.sh: update download link for tar.gz see Update note 5 at https://docs.victoriametrics.com/CHANGELOG.html#v1790	2022-08-30 12:22:42 +03:00
Denys Holius	683039cb63	RELEASE_GUIDE.md: updated the document considering the fixes	2022-08-30 12:21:28 +03:00
Denys Holius	72da5f127a	remove not needed template.json	2022-08-30 12:21:28 +03:00
Denys Holius	ec35ec2125	template.pkr.hcl: fix wrong variable name	2022-08-30 12:19:31 +03:00
Aliaksandr Valialkin	4d5803c16d	docs: fix links to DogStatsD Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3042	2022-08-30 12:19:30 +03:00
Denys Holius	0feec1577c	deployment/docker/Makefile: bump version of Alpine linux to latest 3.16.2 (#3043 ) see https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html	2022-08-30 12:17:13 +03:00
Aliaksandr Valialkin	b26c18a7c5	vendor: `make vendor-update`	2022-08-30 09:45:58 +03:00
Aliaksandr Valialkin	529bcc0761	docs/CHANGELOG.md: document the `044d51b668`	2022-08-30 09:42:51 +03:00
Denys Holius	b71b58b2cf	deployment/docker/Makefile: bump version of Alpine linux to latest 3.16.2 to fix CVE-2022-37434 (#3035 ) see https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html	2022-08-30 09:42:45 +03:00
Bryce Lampe	5f7f1d5aea	Support "HTTP" and "HTTPS" schemes (#3019 ) * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:23:52 +03:00
Aliaksandr Valialkin	8aaaf221cc	app/vmselect/promql: follow-up after `2d71b4859c` - Use getScalar() function for obtaining the expected scalar from phi arg - Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles - Improve the description of this bugfix in the docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026	2022-08-27 01:38:17 +03:00
Dmytro Kozlov	2cf29cc4d1	vmselect/promql: fix panic in histogram_quantiles function (#3029 ) * vmselect/promql: fix panic in histogram_quantiles function * Update docs/MetricsQL.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-08-27 01:34:54 +03:00
Aliaksandr Valialkin	04761419ba	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache This is a follow-up for `c06e7a142c`	2022-08-27 00:29:59 +03:00
Aliaksandr Valialkin	86394b4179	lib/promscrape: optimize discoveryutils.SanitizeLabelName() Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:18:19 +03:00
Zakhar Bessarab	1c7db1c8fd	vmctl: fix progress bar not being stopped on error during import process (#3023 ) vmctl: fix progress bar not being stopped on error during import process	2022-08-26 21:51:20 +03:00
Aliaksandr Valialkin	cead9c1e67	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:09 +03:00
Aliaksandr Valialkin	427d69e775	lib/promrelabel: optimize common regex mismatch cases for `action: replace` and `action: labelmap`	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	da7697fda4	lib/promrelabel: use regexutil.PromRegex for regex matching in actions `labeldrop`,`labelkeep`,`drop` and `keep` This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	e1bd38fa97	lib/promrelabel: optimize matching for commonly used regex patterns in `if` option The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo." and "foo.+" - substring match, e.g. ".foo.*" and ".+foo.+" - alternate values match, e.g. "foo\|bar\|baz"	2022-08-26 14:55:13 +03:00
Aliaksandr Valialkin	c49751adf8	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:43 +03:00
Aliaksandr Valialkin	909e681024	lib/promrelabel: optimize `action: {drop,keep,labeldrop,labelkeep}` with anchored `regex` prefix The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:24:38 +03:00
Aliaksandr Valialkin	d60654eb0a	lib/promrelabel: optimize `action: {labeldrop,labelkeep,keep,drop}` with `regex` containing alternate values For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo\|bar\|baz"	2022-08-24 17:55:54 +03:00
Aliaksandr Valialkin	891eb608df	lib/storage: increase the maximum possible `or` values extracted from regexp from 20 to 100 This should improve time series search speed for regexp filters with big number of `or` values.	2022-08-24 17:16:29 +03:00
Aliaksandr Valialkin	1b14cf18b6	lib/storage: ignore `start text` and `end text` anchors in getOrValues(regexp) function This is OK, since the anchors are implicitly applied to the whole regexp. This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors. For example, `{label="^(foo\|bar)$"}`	2022-08-24 17:16:28 +03:00
Aliaksandr Valialkin	d4cf52f092	app/vmagent: follow-up after `2b22aa1537` - Document the change at docs/CHANGELOG.md - Move auth token parsing from app/vmagent/opentsdbhttp/ to app/vmagent/main.go, since it must be parsed only when multitenancy support is enabled at vmagent side. See https://docs.victoriametrics.com/vmagent.html#multitenancy	2022-08-24 16:21:23 +03:00
Jianyun Cheng	eccae22522	[vmagent] make opentsdb insert url support multitenant (#3015 )	2022-08-24 16:21:22 +03:00
Dmytro Kozlov	d32a6359b0	vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963 ) * vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries * vmselect/promql: cleanup * vmselect/promql: rename config flag * vmselect/promql: add tests * vmselect/promql: use test object instead of log * vmselect/promql: fix posible panic is subquery has more points. add description * vmselect/promql: update tests descriptions * vmselect/promql: update doInternal validation * vmselect/promql: fix linter * vmselect/promql: fix linter * vmselect/promql: update documentation and release notes * wip - Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries. Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries if it was smaller than the -search.maxPointsSubqueryPerTimeseries . - Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag . - Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits . - Update docs/CHANGELOG.md . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-24 15:27:41 +03:00
Aliaksandr Valialkin	e2d8916935	docs: mention that it is safe sharing the collected profiles from security PoV The collected profiles do not contain sensitive information	2022-08-24 14:08:30 +03:00
Aliaksandr Valialkin	7b9ba456ff	app/vmstorage: expose `vm_{hourly,daily}_series_limit_{max,current}_series` metrics if `-storage.max{Hourly,Daily}Series` limits are set These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin	ad6652247c	all: update Google Analytics tracking code from Unversal Analytics to v4 This is needed because Google Analytics devs decided to force their users to update all their tracking codes to GA v4. See https://support.google.com/analytics/answer/9744165 The link to the new tracking property ( G-N9SVT8S3HK ) - https://analytics.google.com/analytics/web/?authuser=1#/a129683199p328513681/admin/streams/table/4004610168	2022-08-24 12:16:18 +03:00
Aliaksandr Valialkin	547d655d29	docs/vmagent.md: fix alerting query when scraped samples are dropped because of exceeded series limit This is a follow-up after `7d26414b2e`	2022-08-24 01:23:37 +03:00
Roman Khavronenko	555a202d80	docs: follow-up after `88425bb285` (#3007 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-24 01:23:37 +03:00
laixintao	76a291a95b	vmalert: add $activeAt into template variables. (#3000 ) vmalert: add `$activeAt` template variable for annotations https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2999	2022-08-24 01:23:37 +03:00
Aliaksandr Valialkin	08b8467e97	app/vmselect/netstorage: make golangci-lint happy by naming the unused padding field as _	2022-08-22 00:32:37 +03:00
Aliaksandr Valialkin	9ddd2699fd	all: remove the remaining bits of io/ioutil The io/ioutil package is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to remove the io/ioutil from source code This is a follow-up for `02ca2342ab`	2022-08-22 00:22:41 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for `02ca2342ab`	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	88e0fe9469	all: use os.ReadDir instead of ioutil.ReadDir The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for `02ca2342ab`	2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin	06f6de6d47	all: use os.{Read\|Write}File instead of ioutil.{Read\|Write}File The ioutil.{Read\|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for `02ca2342ab`	2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin	c5325f2db5	docs/CHANGELOG.md: document `d59d829cdb` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673	2022-08-21 23:37:15 +03:00
Roman Khavronenko	fc2b8b4efd	lib/storage: bump max merge concurrency for small parts to 15 (#2997 ) * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:37:14 +03:00
Ivan Dudin	c7ca124803	Update Cluster-VictoriaMetrics.md (#3005 ) Flags `-search.maxTagKeys` and `-search.maxTagValues` are present at vmstorage and not at vmselect ``` -search.maxTagKeys int -search.maxTagValues int ```	2022-08-21 23:30:15 +03:00
Aliaksandr Valialkin	7f46c17e4e	docs/Cluster-VictoriaMetrics.md: document the best strategies for cluster update / upgrade	2022-08-21 23:20:26 +03:00
Aliaksandr Valialkin	1c7f402598	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the `replacement` field during relabeling For example: - target_label: composite-label replacement: {{source_label1}}-{{source_label2}}	2022-08-21 22:49:24 +03:00
Aliaksandr Valialkin	1509fab1b2	vendor: `make vendor-update`	2022-08-21 19:08:20 +03:00
Aliaksandr Valialkin	ecd8a03b28	docs: change links to Prometheus docs about instant and range queries to links to VictoriaMetrics docs	2022-08-21 19:02:46 +03:00
Aliaksandr Valialkin	3c597d1985	docs/keyConcepts.md: more fixes	2022-08-21 18:53:49 +03:00
Aliaksandr Valialkin	8e86d52c23	docs/FAQ.md: add a question on differences between Grafana Mimir and VictoriaMetrics	2022-08-21 17:00:49 +03:00
Aliaksandr Valialkin	7df5e19de5	docs/keyConcepts.md - clarify docs a bit	2022-08-21 12:00:53 +03:00
Aliaksandr Valialkin	34d764874f	docs/Cluster-VictoriaMetrics.md: clarify required conditions for cluster availability	2022-08-20 09:39:07 +03:00
Aliaksandr Valialkin	6d2354e7a4	app/vmalert/README.md: sync with docs/vmalert.md after `a229182dbe`	2022-08-20 08:54:55 +03:00
laixintao	107ac2a85e	doc: fix broken url of template function for vmalert. (#3002 )	2022-08-20 08:54:54 +03:00
Aliaksandr Valialkin	76844e519b	docs/CHANGELOG.md: document `10402459d8`	2022-08-19 11:44:29 +03:00
Roman Khavronenko	2c59c83191	lib/storage: fix the search for empty label name (#2991 ) * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-19 11:05:09 +03:00
Roman Khavronenko	2256d51418	docs: fix docs formatting related to vmalert (#2994 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-19 11:05:08 +03:00
Denys Holius	fb80735a9a	deployment: bump Grafana version to latest 9.1.0 (#2995 ) see more at https://grafana.com/blog/2022/08/16/grafana-9.1-release/	2022-08-18 12:20:56 +02:00
Aliaksandr Valialkin	84c80b0481	docs/relabeling.md: fix heading for `useful tips for target relabeling`	2022-08-18 01:33:22 +03:00
Aliaksandr Valialkin	26212aa14a	docs/relabeling.md: typo fixes	2022-08-18 01:30:27 +03:00
Aliaksandr Valialkin	570ce15cb8	docs/relabeling.md: improve relabeling tips for scrape targets	2022-08-18 01:24:35 +03:00
Aliaksandr Valialkin	aed6e33cb4	docs/relabeling.md: add a link to VictoriaMetrics enhancements for relabeling	2022-08-18 01:24:33 +03:00
Aliaksandr Valialkin	a6970e3052	docs: fix ordering after adding the docs/relabeling.md	2022-08-18 01:17:55 +03:00
Aliaksandr Valialkin	29eb9d4602	docs/relabeling.md: add a cookbook for common relabeling tasks	2022-08-18 01:14:30 +03:00
Aliaksandr Valialkin	bbfa52bd75	docs: follow-up after `68e56b6fc5`	2022-08-17 21:27:24 +03:00
Aliaksandr Valialkin	2de9d42dcc	go.mod: update github.com/VictoriaMetrics/metrics from v1.22.1 to v1.22.2	2022-08-17 21:27:24 +03:00
Roman Khavronenko	cfcb5ab15b	vmalert: set alert's source link to UI instead of JSON source (#2986 ) We switch default alert's source link to redirect user to vmalert's UI instead of previous JSON object. While it breaks compatibility, it also supposed to improve user's experience. The old behavior can be achieved by updating `-external.alert.source` command-line flag. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-17 21:27:24 +03:00
Aliaksandr Valialkin	d62e9bc964	docs/sd_configs.md: clarify a bit honor_labels config option	2022-08-17 14:24:53 +03:00
Aliaksandr Valialkin	87e0d69bf4	app/vmselect/netstorage: fix a bug introduced in `1a254ea20c` The bug results in `duplicate output time series` error because the same time series is added two times into the orderedMetricNames list inside the tmpBlocksFileWrapper.Finalize(). While at it, properly release all the tmpBlocksFile structs on tbf.Finalize() error. Previously only the remaining tbf entries were released. This could result in resource leak.	2022-08-17 14:07:51 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Roman Khavronenko	0261118765	docs: add new article link (#2989 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	e5c61cac22	docs/CHANGELOG.md: group vmalert features closer to each other	2022-08-17 11:49:53 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	1a363192ff	lib/storage: typo fix in comments after `f830edc0bc`	2022-08-16 13:45:32 +03:00
Aliaksandr Valialkin	dc929e0d16	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when `match[]` filter matches small number of time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:34:23 +03:00
Aliaksandr Valialkin	d39915ff8e	docs/MetricsQL.md: add `the list of supported ... functions` lines just before the corresponding lists This improves the readability a bit	2022-08-16 12:10:09 +03:00
Roman Khavronenko	436b4d90af	docs: update vmalert docs (#2987 ) * mention recently added `$alertID` and `$groupID` variables in the changelog * properly escape template examples in the vmalert's README Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-16 12:10:08 +03:00
Roman Khavronenko	0d5c403b6e	vmalert: support `$alertID` and `$groupID` in template variables (#2983 ) Support of these two variables allows building custom URLs with alert's ID and group ID params. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/517#issuecomment-1207141432 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-16 11:24:01 +03:00
Artem Navoiev	5ae251d074	fix typo in influx guide (#2984 )	2022-08-16 11:23:59 +03:00
Aliaksandr Valialkin	7d7cf2b6fd	app/vmselect: follow-up after `63e0f16062` * Explicitly store a pointer to UserReadableError in the error interface. Previously Go automatically converted the value to a pointer before storing in the error interface. * Add Unwrap() method to UserReadableError, so it can be used transparently with the other code, which calls errors.Is() and errors.As(). * Document the change in docs/CHANGELOG.md	2022-08-15 13:53:19 +03:00
Roman Khavronenko	8a26ec435d	vmselect: introduce UserReadableError type of error (#2894 ) When read query fails, VM returns rich error message with all the details. While these details might be useful for debugging specific cases, they're usually too verbose for users. Introducing a new error type `UserReadableError` is supposed to allow to return to user only the most important parts of the error trace. This supposed to improve error readability in web interfaces such as VMUI or Grafana. The full error trace is still logged with the full context and can be found in vmselect logs. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-15 13:53:18 +03:00
Roman Khavronenko	79e396dac9	docs: mention default vmalert's behavior change in Update notes (#2981 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-15 13:31:59 +03:00
Yury Molodov	dc52b283a3	vmui: shortcut keys legend (#2971 ) * feat: add shortcut modal * feat: add shortcut descriptions * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-15 12:04:30 +03:00
Yurii Kravets	d7dc7c81e7	doc: add upgrade/downgrade without downtime to FAQ (#2973 ) * doc: add upgrade/downgrade without downtime to FAQ Added info on how to upgrade or downgrade VictoriaMetrics without downtime to FAQ Based on reply https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2944#issuecomment-1207502038 * Udate pr Fixing format	2022-08-15 11:31:00 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	1a00c9ef03	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_pod_container_image` label in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin	2fb63dda83	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_service_port_number` label to `role: service` in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin	40c2fae617	vendor: `make vendor-update`	2022-08-15 00:54:50 +03:00
Aliaksandr Valialkin	198d8eeeaa	app/vmalert/templates: add `toTime()` template function in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/10993	2022-08-15 00:49:52 +03:00
Aliaksandr Valialkin	2b58bd9876	lib/promscrape/discovery/dns: add support for resolving MX records See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin	10402459d8	lib/vmselectapi: do not log connection accept/close from vmselect These log messages became too spammy in production clusters after the commit `190c8b463c` , which closes idle connections from vmselect to vmstorage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-08-12 09:15:29 +03:00
Aliaksandr Valialkin	1b39be3305	lib/vmselectapi: add `rpc call` prefix to the trace of the rpc call in order to make it more clear	2022-08-12 00:20:49 +03:00
Aliaksandr Valialkin	78f584fb0b	docs/CHANGELOG.md: clarify the change at `28441711e6`	2022-08-11 23:53:32 +03:00
Roman Khavronenko	36660bcfe2	vmalert: follow-up after `28441711e6` (#2972 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-11 23:52:18 +03:00
Matthew Blewitt	b1b8c99b17	vmalert: mark some url flags as sensitive (#2965 ) Other components, such as `vmagent`, mark these flags as sensitive and hide them from the `/metrics` endpoint by default. This commit adds similar handling to the `vmalert` component, hiding them by default, to prevent logging of secrets inappropriately. Showing of these values is controlled by an additional flag. Follow up to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2947	2022-08-11 23:51:43 +03:00
Aliaksandr Valialkin	1a254ea20c	app/vmselect/netstorage: remove common contention points related to inter-CPU communcations This should improve vmselect performance scalability on systems with many CPU cores. The following tasks were done: - Use separate temporary files for storing the data read from each vmstorage node. This may result in the following potential issues: - Up to N times higher memory usage for performing each query where N is the number of vmstorage nodes known to vmselect. This issue shouldn't increase chances of out of memory errors in most cases, since per-query memory overhead is quite low comparing to the overall vmselect memory usage. - Up to N times higher number of open temporary files where N is the number of vmstorage nodes known to vmselect. This issue should be fixed by increasing the limit on the number of open files. - Use separate counters per each vmstorage node for various stats calculation when reading the data from vmstorage nodes.	2022-08-11 23:22:56 +03:00
Aliaksandr Valialkin	ec3df0b913	app/vmselect/netstorage: improve scalability of blocks processing on systems with multiple CPU cores Previously a single syncwg.WaitGroup was used for tracking the lifetime of processBlock callbacks across all the per-vmstorage goroutines. This could be slow on systems with many CPU cores because of inter-CPU synchronization overhead. Use a separate per-vmstorage sync.WaitGroup instead in order to reduce inter-CPU synchronization overhead. This should imrpove performance for heavy queries over big number of blocks on multi-CPU systems.	2022-08-11 21:37:24 +03:00
Aliaksandr Valialkin	690b505975	deployment/docker: specify docker image tags for all the docker images for reproducible `docker-compose up` runs	2022-08-09 12:30:22 +03:00
Roman Khavronenko	f42853275f	lib/storage: prevent excessive loops when storage is in RO (#2962 ) * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:47 +03:00
Roman Khavronenko	f90f654cf2	vmalert: sort groups at `/alerts` page (#2968 ) Sorting will produce deterministic output of grops on the page. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-09 12:08:54 +03:00
Aliaksandr Valialkin	91d0b45537	docs/CHANGELOG.md: cut v1.80.0	2022-08-08 19:59:02 +03:00
Aliaksandr Valialkin	7e892d3322	docs/CHANGELOG.md: add changes for v1.79.2	2022-08-08 18:00:38 +03:00
Aliaksandr Valialkin	e0865f4174	vendor: update github.com/VictoriaMetrics/metrics from v1.21.0 to v1.22.1	2022-08-08 17:19:16 +03:00
Aliaksandr Valialkin	b8877da35b	docs/CHANGELOG.md: link to the issue related to vmselect panic in multi-level cluster setup Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961	2022-08-08 15:09:39 +03:00
Aliaksandr Valialkin	ceb158c5bb	docs/vmagent.md: typo fix in __meta_kubernetes_annotation_prometheus_io_tenant label name It must be __meta_kubernetes_pod_annotation_prometheus_io_tenant	2022-08-08 14:53:28 +03:00
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after `2c553d5a2f` - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950 ) * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Aliaksandr Valialkin	69d62d5736	docs: sync -help output with the latest changes	2022-08-08 14:05:36 +03:00
Roman Khavronenko	f31132b70b	lib/promrelabel: fix expected test result (#2957 ) follow-up after `68c4ec9472` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:48:14 +03:00
Aliaksandr Valialkin	221dd3a224	all: bump the minimum supported version of Go from 1.17 to 1.18 This is needed because some dependencies uses generics, which have been appeared in Go1.18 This is a follow-up for `caf3dd4fa2`	2022-08-08 13:45:39 +03:00
Roman Khavronenko	6af40f6275	vmalert: remove notions of vmalert being compatible with VM only (#2954 ) vmalert can be successfully used with datasources compatible with Prometheus HTTP API. So we remove comments or notes in Readme which are saying opposite. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:44:15 +03:00
Aliaksandr Valialkin	1996e36cf0	app/vmselect/netstorage: prevent from calling processBlocks callback after the exit from ProcessBlocks function This should prevent from panic at multi-level vmselect when the top-level vmselect is configured with -replicationFactor > 1	2022-08-08 13:32:44 +03:00
Roman Khavronenko	3c583c16a1	dashboards: add `Cache usage %` panel to Caches row (#2960 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2941 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 11:45:17 +02:00
Aliaksandr Valialkin	12f4c89ff7	docs/CHANGELOG.md: link to the issue with improper handling of `enpoint` option at ec2_sd_configs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2917	2022-08-08 03:33:17 +03:00
Aliaksandr Valialkin	9039f23bd1	lib/promrelabel: do not split regex into multiple lines if it contains groups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:16:15 +03:00
Aliaksandr Valialkin	23b7fa897a	docs/CHANGELOG.md: link to the issue regarding the increased load on Consul This is a follow-up for `68de1f4e4a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-08 02:23:01 +03:00
Aliaksandr Valialkin	b5a05f627c	docs/_includes/img.html: fix https://github.com/VictoriaMetrics/VictoriaMetrics/security/code-scanning/28	2022-08-08 01:04:20 +03:00
laixintao	9f5fc040a7	bugfix: fix vmalert navbar url. (#2949 ) the doc url should not be joined by `prefix` because it's an abs url.	2022-08-08 00:29:48 +03:00
Aliaksandr Valialkin	2635211bf4	app/vmselect/netstorage: properly detect and log timeout errors when querying vmstorage from vmselect This change is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2937 Thanks to @isodude for the initial pull request.	2022-08-08 00:21:05 +03:00
Yury Molodov	a2ecf311b8	vmui: graph action on moush hold and move (#2915 ) * fix: change event for graph panning * fix: change detect key * feat: add zoom in with mouse selection * - document the change - run `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-07 23:56:08 +03:00
Aliaksandr Valialkin	a17030090b	lib/auth: follow-up after `b6a6a659f4`	2022-08-07 23:15:25 +03:00
Dmytro Kozlov	a266e3e136	lib/auth: add tests for NewToken function (#2921 ) * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:15:23 +03:00
Aliaksandr Valialkin	fd1ac20760	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:41 +03:00
Aliaksandr Valialkin	7a1a95fd3e	docs/sd_configs.md: mention that http client configs can contain `headers` and `proxy_headers` options	2022-08-07 22:41:02 +03:00
Aliaksandr Valialkin	6b49958e04	vendor: `make vendor-update`	2022-08-07 22:38:32 +03:00
Aliaksandr Valialkin	bfa490e921	Makefile: update golangci-lint from v1.47.1 to v1.48.0 This is needed for adding support for Go 1.19	2022-08-07 22:33:40 +03:00
Aliaksandr Valialkin	f28f4ad551	deployment/docker: update Go builder from Go1.18.5 to Go1.19.0 See https://tip.golang.org/doc/go1.19 Notable changes: * GOMEMLIMIT environment variable - see https://tip.golang.org/doc/gc-guide * Faster CPU profiler * Faster sort algorithm	2022-08-07 21:31:20 +03:00
Aliaksandr Valialkin	573f6c8d6c	Makefile: remove redundant `-mod=vendor` option when running Go tools The `-mod=vendor` is automatically set when there is a `vendor` directory starting from Go1.14 - see https://go.dev/doc/go1.14#go-command Since the minimum supported Go version for VictoriaMetrics is Go1.17, then the `-mod=vendor` option is no longer needed.	2022-08-07 20:40:41 +03:00
Aliaksandr Valialkin	ed891049d5	docs/sd_configs.md: mention that http client options can be specified in scrape_configs section	2022-08-07 00:20:28 +03:00
Aliaksandr Valialkin	21f588e9cf	app/vmselect/promql: fix TestVmrangeBucketsToLE test after `8516670582` Remove 'AccountID=0, ProjectID=0, ' prefix for cluster version	2022-08-07 00:15:10 +03:00
Aliaksandr Valialkin	5e43145031	docs: add docs for scrape_configs section	2022-08-07 00:04:55 +03:00
Aliaksandr Valialkin	0596bae703	docs/sd_configs.md: add docs for static_configs	2022-08-06 23:17:28 +03:00
Aliaksandr Valialkin	d4c3702afe	docs/sd_configs.md: add docs for openstack_sd_configs	2022-08-06 23:07:11 +03:00
Aliaksandr Valialkin	659f093d87	docs/sd_configs.md: document kubernetes_sd_configs	2022-08-06 22:39:16 +03:00
Aliaksandr Valialkin	77bd4e37cc	lib/promscrape/discovery/kubernetes: add missing `__meta_kubernetes_ingress_class_name` label for `role: ingress` See `7e65ad3e43` and `7e1111ff14`	2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin	98deef04f5	docs/sd_configs.md: document http_sd_configs	2022-08-05 19:50:29 +03:00
Aliaksandr Valialkin	8d63e316a9	docs/sd_configs.md: document gce_sd_configs	2022-08-05 19:37:23 +03:00
Aliaksandr Valialkin	ecdd57e13b	docs/sd_configs.md: document file_sd_configs	2022-08-05 19:17:29 +03:00
Aliaksandr Valialkin	6d5c8f82cf	docs/sd_configs.md: document eureka_sd_configs	2022-08-05 19:05:33 +03:00
Aliaksandr Valialkin	091c654c92	docs/sd_configs.md: document ec2_sd_configs	2022-08-05 18:52:39 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom `endpoint` option in ec2_sd_configs This option was ignored since `d289ecded1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	8d6ac974b2	docs/sd_configs.md: add docs for dockerswarm_sd_configs	2022-08-05 16:20:30 +03:00
Aliaksandr Valialkin	80ecfcf759	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin	d02ecc71bb	docs/sd_configs.md: document docker_sd_configs	2022-08-05 15:36:52 +03:00
Aliaksandr Valialkin	a931f75d51	docs/sd_configs.md: document dns_sd_configs	2022-08-05 15:27:07 +03:00
Aliaksandr Valialkin	81dc3a34ed	docs/sd_configs.md: document digitalocean_sd_configs	2022-08-05 15:17:09 +03:00
Aliaksandr Valialkin	c4e6973bc8	docs/sd_configs.md: document consul_sd_configs	2022-08-05 15:04:06 +03:00
Aliaksandr Valialkin	85b04732ed	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin	2c2a5ff553	docs/sd_configs.md: document azure_sd_configs	2022-08-05 14:16:10 +03:00
Aliaksandr Valialkin	17290a4598	lib/promscrape/discovery/yandexcloud: further code cleanup after `83a4abda3f`	2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin	df34f8c5b8	docs: fixes after `83a4abda3f`	2022-08-05 10:15:28 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after `6e5ac32fba` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923 ) * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	a07891cca6	docs: exporing -> exporting typo fix This is a follow-up after `ccb6cb6501`	2022-08-04 20:31:59 +03:00
kevinflynn387	6d909c87f9	Docs: Fix typo from "HA paris" to "HA pairs" (#2935 ) * Update README.md: fix typo to "HA pairs" Update README.md to fix typo from "HA paris" to "HA pairs" * Update docs/README.md : Fix of typo from "HA paris" Update docs/README.md : Fix of typo from "HA paris" to "HA pairs" * Update of Single-server-VictoriaMetrics.md to fix typo "HA paris" to "HA pairs" Update of Single-server-VictoriaMetrics.md to fix typo "HA paris" to "HA pairs"	2022-08-04 20:29:02 +03:00
Aliaksandr Valialkin	dc58cb3fd8	app/vminsert/netstorage: pre-initialize the remaining throttled loggers This is a follow-up after `6c66804fd3`	2022-08-04 18:34:42 +03:00
Aliaksandr Valialkin	43185353bc	app/vmselect/netstorage: cleanup after `92630c1ab4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	db049fed84	lib/mergeset: cleanup after `de6dd1cd5a` Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	568e4aac5e	vendor: update github.com/VictoriaMetrics/metrics from v1.20.1 to v1.21.0 This adds the following push-related metrics when -pushmetrics.url is set: - metrics_push_interval_seconds - metrics_push_total - metrics_push_errors_total - metrics_push_bytes_pushed_total - metrics_push_duration_seconds - metrics_push_block_size_bytes Updates https://github.com/VictoriaMetrics/metrics/issues/35	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	98caffcfc4	docs: fix the recommended url for -vmalert.proxyURL accroding to `8667307d73`	2022-08-04 18:03:24 +03:00
Aliaksandr Valialkin	ba3ca5d1cd	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:43:24 +03:00
Aliaksandr Valialkin	64183d4416	docs/CHANGELOG.md: document v1.79.1 security fix	2022-08-02 13:31:14 +03:00
Denys Holius	9b4024cd62	deployment/docker/Makefile: added docker-scan (#2916 ) * deployment/docker/Makefile: added docker-scan docker-scan based on native 'docker scan' function that use snyk.io, see https://docs.docker.com/engine/scan/ * set to call 'docker-scan after release binaries but before publishing	2022-08-02 13:26:43 +03:00
Aliaksandr Valialkin	f2e56f0dfd	deployment/docker: update Go builder from v1.18.4 to v1.18.5 See https://github.com/golang/go/issues?q=milestone%3AGo1.18.5+label%3ACherryPickApproved	2022-08-02 13:13:11 +03:00
Aliaksandr Valialkin	a1e49606ed	app/{vmselect,vmalert}: properly generate http redirects if `-http.pathPrefix` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin	7adc5a461b	vendor: `make vendor-update`	2022-08-02 09:20:03 +03:00
Aliaksandr Valialkin	9f95099cf4	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:13:41 +03:00
Aliaksandr Valialkin	c81d2b4c18	app/vmselect/netstorage: initializes tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths	2022-07-30 00:39:14 +03:00
Aliaksandr Valialkin	5ddae2e293	app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel This should reduce CPU usage needed for rand.Rand initialization	2022-07-30 00:31:00 +03:00
Aliaksandr Valialkin	f1a49fe779	docs/Release-Guide.md: document that feature docs must be updated with the release where the feature appeared	2022-07-30 00:16:34 +03:00
Aliaksandr Valialkin	586d267a44	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:11:14 +03:00
Aliaksandr Valialkin	962ed46583	lib/mergeset: optimize mergeInmemoryBlocks() function Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin	3bbe9054d3	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin	547cb1edce	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin	5f2b5bd173	lib/mergeset: add tests and benchmarks for commonPrefixLen function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:25:23 +03:00
Dmytro Kozlov	ab645b5fab	vmselect/promql: add tests for vmrangeBucketsToLE (#2907 ) * vmselect/promql: add tests for vmrangeBucketsToLE * vmselect/promql: cleanup * vmselect/promql: cleanup * vmselect/promql: fix panic tests want result * vmselect/promql: cleanup * vmselect/promql: update test name * vmselect/promql: fix linter error * vmselect/promql: refactor testcases * vmselect/promql: cleanup * vmselect/promql: remove unused reassign to workers, fix typo * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-26 20:43:43 +03:00
Aliaksandr Valialkin	749e825020	lib/pushmetrics: `make fmt`	2022-07-26 20:41:23 +03:00
Aliaksandr Valialkin	8722342819	docs/CHANGELOG.md: clarify docs about the ability to configure multiple sections for `vmauth` with identical `username` and different `password` values	2022-07-26 19:39:00 +03:00
Aliaksandr Valialkin	86e0abe357	docs/Single-server-VictoriaMetrics.md: improve "Push metrics" docs	2022-07-26 19:34:29 +03:00
Aliaksandr Valialkin	9f1e558c58	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin	8a83c59956	app/vmselect/promql: reduce the diff for `f148cffc8a` This is a follow-up for `c826f06366`	2022-07-26 19:21:16 +03:00
Aliaksandr Valialkin	27c4c5a530	vendor: `make vendor-update`	2022-07-25 10:54:54 +03:00
Aliaksandr Valialkin	bf797ba4d9	all: push metrics to -pusmetrics.url in gzip-compressed form in order to reduce the needed network bandwidth	2022-07-25 10:54:54 +03:00
Alan Liang	f97de473f4	vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902 ) Co-authored-by: alanwzliang <alanwzliang@tencent.com>	2022-07-25 10:49:48 +03:00
Aliaksandr Valialkin	764fc04756	app/vmalert: fix after `da10962d4c`	2022-07-25 09:48:41 +03:00
Roman Khavronenko	da10962d4c	vmselect: cover special cases for vmalert's routing in single-node version (#2845 ) * vmselect: cover special cases for vmalert's routing in single-node version * remove trailing `/` from requests * redirect to vmalert's home page when `/vmalert` is requested. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: fix review comments Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update app/vmselect/main.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-25 09:43:44 +03:00
Aliaksandr Valialkin	5d62f5a324	app/vmalert/config: add missing docs for ValidateTplFn	2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin	3d4c312ba2	app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system Reduce inter-CPU communications when processing the query over big number of time series. This should improve performance for queries over big number of time series on systems with many CPU cores. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896 Based on `b596ac3745` Thanks to @zqyzyq for the idea.	2022-07-25 09:22:28 +03:00
Roman Khavronenko	970f36de17	vmalert: remove `notifier` dependency from `config` (#2906 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	1b5799f894	lib/promscrape: set `up=0` for partially failed scrape in stream parsing mode This behaviour aligns with Prometheus behavior	2022-07-22 13:38:25 +03:00
Roman Khavronenko	01755fac38	vmalert: remove dependency on datasource pkg from config (#2905 ) * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	b1129fda0c	docs/vmgateway: typo fix in regexp filter: `~=` is substituted with `=~`	2022-07-21 22:37:42 +03:00
Aliaksandr Valialkin	cfeb6138ae	docs/BestPractices.md: do not mention NFS Though VictoriaMetrics works OK on NFS-like filesystems, it is not a best practice	2022-07-21 21:31:14 +03:00
Aliaksandr Valialkin	301a27b9f8	vendor: `make vendor-update`	2022-07-21 21:11:58 +03:00
Aliaksandr Valialkin	59bb42be06	app/vmselect/vmui: `make vmui-update` after `edecd2493c`	2022-07-21 21:00:36 +03:00
Aliaksandr Valialkin	aae522e501	docs/CHANGELOG.md: document bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874 This is a follow-up for `edecd2493c`	2022-07-21 21:00:34 +03:00
Aliaksandr Valialkin	3cbe232186	docs/CHANGELOG.md: follow-up after `88edb3f6cf`	2022-07-21 20:53:24 +03:00
Aliaksandr Valialkin	b49fc2f9f3	app/vmalert/utils: add missing docs to WithHeaders func added at `70a822f3a0`	2022-07-21 20:49:38 +03:00
Roman Khavronenko	d0abdc2b5b	vmalert: allow configuring custom headers per group (#2901 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:48:05 +03:00
Roman Khavronenko	356d8b99e0	vmalert: allow configuring custom headers for URLs (#2897 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:45:12 +03:00
dependabot[bot]	272f207afc	build(deps): bump terser in /app/vmui/packages/vmui (#2895 ) Bumps [terser](https://github.com/terser/terser) from 5.13.1 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-07-21 20:42:01 +03:00
Yury Molodov	d1b88f3b5d	fix: change the z-index of the datepicker (#2891 )	2022-07-21 20:41:16 +03:00
Roman Khavronenko	04c4f8bafd	vmselect: return correct error for second part of expression (#2893 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:40:47 +03:00
Aliaksandr Valialkin	db96221cee	docs/Cluster-VictoriaMetrics.md: update after `fe68bb3ba7`	2022-07-21 20:21:39 +03:00
Aliaksandr Valialkin	fe68bb3ba7	all: follow-up after `46f803fa7a` Add -pushmetrics.* command-line flags to all the VictoriaMetrics apps	2022-07-21 20:18:25 +03:00
Aliaksandr Valialkin	f00a6bf837	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin	536ee9fc18	deployment/docker: update alpine base image from 3.16.0 to 3.16.1 See https://alpinelinux.org/posts/Alpine-3.16.1-released.html	2022-07-19 19:51:36 +03:00
Aliaksandr Valialkin	7a06ee11fd	docs/vmauth.md: mention that multiple recrods for the same username are supported This is a follow-up for `88029c521c`	2022-07-19 19:43:20 +03:00
Nikolay	0102eb3869	app/vmauth: allow duplicate usernames (#2888 ) Usernames could be duplicate if it has uniq password. vmauth makes routing based on auth token and username + password combination must be unique for this case.	2022-07-19 19:35:28 +03:00
Denys Holius	d635169c90	Update golangci version to latest v1.47.1 (#2890 ) See https://github.com/golangci/golangci-lint/releases/tag/v1.47.1	2022-07-19 19:31:16 +03:00
Denys Holius	419f913bb9	docs: faq update (#2889 ) * docs: faq update add links for OpenBSD ports & pre built binaries * Update docs/FAQ.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-19 17:51:04 +03:00
Aliaksandr Valialkin	f992f96a88	app/vmselect/promql: execute `q1` and `q2` from `q1 op q2` in parallel if labels pushdown cannot be applied This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886	2022-07-19 14:29:41 +03:00
Aliaksandr Valialkin	69c1272d1b	docs/Articles.md: add a link to "How do We Keep Metrics for a Long Time in VictoriaMetrics" talk	2022-07-19 13:01:39 +03:00
Aliaksandr Valialkin	2d1366353c	lib/promscrape: reload all the scrape configs when the `global` section is changed inside `-promscrape.config` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:42 +03:00
Roman Khavronenko	23e85e0fc5	vmagent: expose metric `vmagent_remotewrite_queues` (#2871 ) The new metric `vmagent_remotewrite_queues` exports a static value of number of configured remote write queus. This metric is useful to calculate total saturation per each configured URL with given number of queues. See corresponding changes to vmagent alerts and dashboard. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-18 14:41:04 +03:00
Aliaksandr Valialkin	8b5b64856b	deployment/docker: update Grafana from v9.0.2 to v9.0.3 See https://grafana.com/blog/2022/07/14/grafana-v9-0-3-8-5-9-8-4-10-and-8-3-10-released-with-high-severity-security-fix/	2022-07-18 14:27:42 +03:00
Aliaksandr Valialkin	d920b0afec	app/vmselect/promql: properly return `q1` series from `q1 ifnot q2` when `q2` returns nothing	2022-07-18 14:24:23 +03:00
Aliaksandr Valialkin	f933d5f142	app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results	2022-07-18 14:24:22 +03:00
Aliaksandr Valialkin	83fef7daf5	docs/CHANGELOG.md: document `2f9668eba5`	2022-07-18 12:37:34 +03:00
Boris Petersen	61e5f89cfb	fix assume role when running in ECS. (#2876 ) This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:37:33 +03:00
Aliaksandr Valialkin	979444b4ed	all: fix other typos in the same way as `6f4d9b2a48` does	2022-07-18 12:10:41 +03:00
cui fliter	e0d30f6ec5	fix some typos (#2882 ) Signed-off-by: cui fliter <imcusg@gmail.com>	2022-07-18 12:10:40 +03:00
Aliaksandr Valialkin	b2d9379bd0	docs/CHANGELOG.md: mention that v1.79.0 changes binary names to `$(APP_NAME)-$(GOOS)-$(GOARCH)...` naming	2022-07-18 12:00:45 +03:00
zhenyuxie	14c6212a61	fix inmemoryBlock's Less method (#2881 )	2022-07-18 12:00:45 +03:00
Aliaksandr Valialkin	059e769674	vendor: `make vendor-update`	2022-07-18 12:00:45 +03:00
Aliaksandr Valialkin	7669d34931	docs/Single-server-VictoriaMetrics.md: add a link on docs how to send signals to processes in Linux	2022-07-18 10:59:28 +03:00
Aliaksandr Valialkin	fae54e2e56	docs/Troubleshooting.md: clarify some texts there	2022-07-15 12:23:39 +03:00
Aliaksandr Valialkin	0237d987c0	docs/Troubleshooting.md: mention about `-search.setLookbackToStep` command-line flag in the section explaining why `/api/v1/query_range` returns calculated data	2022-07-15 11:55:34 +03:00
Aliaksandr Valialkin	0d4676f313	docs/Single-server-VictoriaMetrics.md: mention that the delete API doesnt delete the associated entries from inverted index	2022-07-14 21:29:28 +03:00
Aliaksandr Valialkin	a87f143e92	docs/CHANGELOG.md: cut v1.79.0	2022-07-14 16:15:35 +03:00
Roman Khavronenko	80de6face8	vmalert: drop support of deprecated `extra_filter_labels` param (#2870 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-14 16:10:46 +03:00
Aliaksandr Valialkin	36062d27c5	docs/CHANGELOG.md: formatting fixes for update notes	2022-07-14 12:48:17 +03:00
Aliaksandr Valialkin	70b9925bf7	app: fix `make publish-*` after `ed93330e66` Add missing `-linux` substring to built binary names for copying into Docker images	2022-07-14 11:01:34 +03:00
Aliaksandr Valialkin	e6ee940daf	docs: update descriptions for command-line flags according to the latest changes	2022-07-14 00:57:44 +03:00
Yury Molodov	1ca20caa4b	vmui: optimize table view (#2867 ) * feat: optimize table view * fix: add column display setting * app/vmselect: `make vmui-update` Also document the change at docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-14 00:16:57 +03:00
Dmytro Kozlov	99402541fb	vmui: update time picker behavior (#2847 ) * vmui: update time picker behavior * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-14 00:06:22 +03:00
Yury Molodov	23c3b5f8c7	vmui: change selection from autocomplete (#2862 ) * fix: change selection from autocomplete * update docs/CHANELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:56:04 +03:00
Nikolay	c007b129cb	lib/promscrape: adds azure service discovery (#2743 ) * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:45:43 +03:00
Boris Petersen	cbf01ee4ae	fix typo introduced in pr #2604 (#2866 ) Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-13 23:45:42 +03:00
Aliaksandr Valialkin	83221bc480	deployment/docker: update Go builder from go1.18.3 to go1.18.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.18.4+label%3ACherryPickApproved	2022-07-13 20:31:54 +03:00
Aliaksandr Valialkin	da6c85a2f6	all: follow-up for `d99ba3481b`	2022-07-13 17:17:08 +03:00
Dmytro Kozlov	4e4def9df8	Rename release packages (#2810 ) * makefile: add os to each release file * makefile: update vmutils arm64 * makefile: update victoria-metrics release process * makefile: update publish with os * makefile: update publish with os * makefile: change tar library * update release logic * copy all releases * sort command by GOOS * rollback commands * rollback OSARCH * fix commands * cleanup * fix windows build * sort build by GOOS, update README.md	2022-07-13 17:11:01 +03:00
Aliaksandr Valialkin	8d33eee7d6	vendor: `make vendor-update`	2022-07-13 17:10:57 +03:00
Aliaksandr Valialkin	be5fb139d6	docs/CHANGELOG.md: document `91faa152a5`	2022-07-13 12:41:25 +03:00
guidao	f2d24a660b	add next retention metric (#2863 ) Co-authored-by: wangfeng <wangfeng@zhihu.com>	2022-07-13 12:41:22 +03:00
Aliaksandr Valialkin	919939ee9f	app/vmselect/promql: consistency update after `93fbd0c54b`	2022-07-13 12:34:28 +03:00
Dmytro Kozlov	5256af2291	lib/mergeset: fix linter error (#2864 )	2022-07-13 12:34:28 +03:00
Roman Khavronenko	214e01499c	promql: return `step` as scrapeInterval when it can't be calculated (#2865 ) The change allows to specify default value for `getScrapeInterval` function when actual interval can't be calculated. Before the change, function were returning `maxSilenceInterval` (5m) in such cases, which may be not correct for instant queries processing. The specific scenario where using `maxSilenceInterval` caused issues is the following: 1. Series becomes stale; 2. Client (in this case vmalert) continues to request series every 15s; 3. Database returns empty results as expected; 4. But at some specific moment of time database returns datapoints from `now()-5m`, because lookback window was extended to `maxSilenceInterval`. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin	205c2a468a	docs/CHANGELOG.md: mention that the communication protocol between `vmselect` and `vmstorage` nodes is updated in the new release	2022-07-13 12:16:59 +03:00
Denys Holius	083af28cf8	deployment/docker/docker-compose.yml: update Grafana from v8.5.1 to v9.0.2 (#2859 ) See https://grafana.com/blog/2022/06/14/grafana-9.0-release-oss-and-cloud-features	2022-07-12 20:03:28 +03:00
Aliaksandr Valialkin	5867708b3d	app/vmselect/promql: validate function name before evaluating its arguments This avoids unneeded evaluation of args for unknown functions	2022-07-12 19:49:54 +03:00
Roman Khavronenko	a42063909f	alerts: correct expression for `DiskRunsOutOfSpaceIn3Days` (#2856 ) The negative value for ETA can happen when deduplication is enabled and `rate` over `vm_deduplicated_samples_total` becomes bigger than actual ingestion rate. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-12 14:14:47 +02:00
Aliaksandr Valialkin	7cbcbea49d	lib/mergeset: optimize merge speed a bit Use heap.Fix instead of heap.Pop + heap.Push when merging blocks	2022-07-12 12:52:36 +03:00
Aliaksandr Valialkin	fbb403b5c0	app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples	2022-07-12 12:30:24 +03:00
Aliaksandr Valialkin	aee08117e9	app/vmselect/netstorage: add mergeSortBlocks benchmark for the worstcase	2022-07-12 12:26:27 +03:00
Aliaksandr Valialkin	03680b9334	docs: add links to https://docs.victoriametrics.com/CHANGELOG.html in relevant docs	2022-07-11 21:23:51 +03:00
Aliaksandr Valialkin	b95db7045d	docs/Articles.md: add a link to https://habr.com/ru/company/sravni/blog/672908/	2022-07-11 21:10:59 +03:00
Aliaksandr Valialkin	dd2b1f9ec4	app/vmselect/prometheus: follow-up after `3efe33b917`	2022-07-11 20:37:21 +03:00
Dmytro Kozlov	6734d77d97	vmselect/prometeus: Add limit param to api/v1/series api endpoint (#2851 ) * issue-2841: Add limit param to api/v1/series api endpoint * issue-2841: add change log * issue-2841: update logic * issue-2841: simplify logic * issue-2841: simplify logic, add information to documentation	2022-07-11 20:37:20 +03:00
Aliaksandr Valialkin	ed2126aec3	app/vmselect: follow-up after `6dda254e01`	2022-07-11 19:48:18 +03:00
Roman Khavronenko	6dda254e01	vmselect: cover special cases for vmalert's routing in cluster version (#2844 ) * vmselect: cover special cases for vmalert's routing * remove trailing `/` from requests * redirect to vmalert's home page when `/vmalert` is requested. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmselect: fix review comments Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update app/vmselect/main.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-11 19:33:33 +03:00
Aliaksandr Valialkin	eab8ebbe11	all: `make fmt` via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin	3662ce8323	docs: make more clear the relation between replication and deduplication This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2852	2022-07-11 19:23:24 +03:00
Aliaksandr Valialkin	9a06353091	vendor: `make vendor-update`	2022-07-11 18:15:37 +03:00
Aliaksandr Valialkin	c0af52228a	app/vmselect/netstorage: add benchmarks for mergeSortBlocks This is a follow-up for `743ff84863`	2022-07-11 12:53:46 +03:00
Aliaksandr Valialkin	d442ee4610	app/vmselect/netstorage: optimize mergeSortBlocks function - Use binary search instead of linear scan when locating the run of smallest timestamps in blocks with intersected time ranges. This should improve performance when merging blocks with big number of samples - Skip samples with duplicate timestamps. This should increase query performance in cluster version of VictoriaMetrics with the enabled replication.	2022-07-09 00:35:38 +03:00
Roman Khavronenko	78da074ad3	metricsql: properly evaluate `timezone_offset` over time interval (#2842 ) * metricsql: properly evaluate `timezone_offset` over time interval https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-08 14:04:40 +03:00
Aliaksandr Valialkin	64419aa97c	app/vmalert/utils/links.go: document Prefix function, which has been added in `b29fafa86b`	2022-07-08 13:25:23 +03:00
Aliaksandr Valialkin	decbf84e55	app/vmselect/vmui: follow-up after `0bf6841140` * Document the bugfix in docs/CHANGELOG.md * Run `make vmui-update` for updating static js files for vmui, which are included into vmselect	2022-07-08 13:22:57 +03:00
Dmytro Kozlov	5afe15f53a	vmui: fix query for json and table tabs (#2846 ) * vmui: fix query for json and table * vmui: add step param	2022-07-08 13:22:55 +03:00
Aliaksandr Valialkin	d0a943c487	docs/CHANGELOG.md: clarifications after `e9b977859b`	2022-07-08 13:05:11 +03:00
Aliaksandr Valialkin	a5588bc6f4	docs/CHANGELOG.md: recommend clearing caches after the upgrade from v1.78.0 to v1.78.1	2022-07-08 13:05:11 +03:00
Roman Khavronenko	b29fafa86b	vmalert: deprecate alert's status link (#2840 ) * vmalert: deprecate alert's status link Deprecate alert's status link `/api/v1/<groupID>/<alertID>/status` in favour of `api/v1/alerts?group_id=<group_id>&alert_id=<alert_id>"`. The change was needed for simplifying logic in vmselect for proxying vmalert's requests. The old alert's status link will be still supported for a few versions but will be removed in the future. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: fix review comments Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-08 13:05:11 +03:00
Aliaksandr Valialkin	9ccae66337	docs/CHANGELOG.md: link to the issue related to multi-level vmselect	2022-07-08 05:23:51 +03:00
Aliaksandr Valialkin	e855744871	docs/CHANGELOG.md: recommend upgrading from v1.78.0 to v1.78.1 because of the bug in v1.78.0	2022-07-08 03:45:06 +03:00
Aliaksandr Valialkin	5a5c8534d1	docs/CHANGELOG.md: cut v1.78.1	2022-07-08 01:03:04 +03:00
Aliaksandr Valialkin	69718a6c16	docs/vmagent.md: clarify that vmagent supports Prometheus-compatible service discovery	2022-07-07 21:22:02 +03:00
Aliaksandr Valialkin	2a53b8961e	docs/vmagent.md: typo fixes after `ef2eeeb642`	2022-07-07 21:17:13 +03:00
Aliaksandr Valialkin	b5a272ae5d	vendor: `make vendor-update`	2022-07-07 20:49:36 +03:00
Aliaksandr Valialkin	372ecc5c0e	docs/vmagent.md: actualize docs	2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin	1ff2b8ed0e	docs/Single-server-VictoriaMetrics.md: actualize `vmui` docs according to recent changes	2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin	d8483f599e	docs/CHANGELOG.md: typo fixes	2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin	3d0ce901cb	docs: sync after recent changes	2022-07-07 02:44:24 +03:00
Aliaksandr Valialkin	f03d4b2aa2	docs/CHANGELOG.md: link another bugreport related to the bug with per-day inverted index in v1.78.0	2022-07-07 02:35:35 +03:00
Aliaksandr Valialkin	5794886662	lib/promscrape: properly set Host header when sending requests via http proxy Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin	9dd5d3a431	docs/CHANGELOG.md: link to the issue about incorrect per-day index handling in v1.78.0	2022-07-07 02:00:58 +03:00
Aliaksandr Valialkin	95add1e8e4	app/{vmagent,vminsert}: follow-up after `d19e46de55` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:32:11 +03:00
Pedro Gonçalves	af5f967307	app/vminsert: allow to ingest datadog metrics with simpler tags - not enforcing key:value (#2839 )	2022-07-07 01:32:10 +03:00
Roman Khavronenko	218dfe7956	docs: mention deduplication issue for HA vmalert topology (#2838 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-07 01:13:56 +03:00
Aliaksandr Valialkin	4d03ac90fc	lib/promscrape/discovery/kubernetes: properly populate service-level labels for `role: endpointslice` targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin	c4cc45d7f8	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to `role: endpoints` and `role: endpointlice` targets in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/10759	2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin	6bb4bcd30c	app/vmui: tune visual presentation of trace view	2022-07-06 14:16:24 +03:00
Aliaksandr Valialkin	eee384c9c9	app/vmselect: add `-clusternative.tls*` options for mTLS setup in multi-level clusters	2022-07-06 13:48:21 +03:00
Aliaksandr Valialkin	f9303e494c	lib/promscrape: fix a test after `c66f676f3b`	2022-07-06 13:25:17 +03:00
Aliaksandr Valialkin	195dccf678	app/vmselect: add ability to query `vmselect` from another `vmselect`	2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin	ea4ab0df36	docs: sync with single-node branch after `ed89106274`	2022-07-06 13:02:05 +03:00
Aliaksandr Valialkin	714958deab	all: follow-up after `ed89106274`	2022-07-06 12:48:52 +03:00
Roman Khavronenko	56f4058fe3	vmalert: make UI and assets links relative (#2831 ) * make all links in vmalert relative, so links continue to work even if vmalert sits behind the proxy; * update vmalert's routing to always have component-unique path prefix, e.g. /vmalert; See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-06 12:47:53 +03:00
Aliaksandr Valialkin	498c6d6e72	lib/promscrape: push `scrape_samples_limit` metric to remote storage if `sample_limit` option is set in `scrape_config` for this target See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	cdd89d9cc2	app/vmselect: properly generate response for /api/v1/series The response has been broken in `7d5d33fd71`	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	b4489028f3	lib/storage: typo fix in MetricName.Unmarshal error	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	1ec4dfd678	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics This reduces the number of args to vmselectapi calls	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	2e721f7d16	lib/vmselectapi: rename Server.MustClose to more clear Server.MustStop	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	270e555f47	lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	f4df43f7cc	app/vmselect/netstorage: remove unused auth.Token arg	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	78eeca6f0d	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	5afa54e845	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() This improves the API consistency	2022-07-06 12:46:21 +03:00
Roman Khavronenko	f51bc07d97	vmselect: allow proxying requests to vmalert (#2833 ) The change allows to proxy requests with prefix `/vmalert` to the vmalert component if `-vmalert.proxyURL` is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825 and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2831 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-06 10:47:51 +02:00
Aliaksandr Valialkin	78f9a8aafd	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit `55e7afae3a`, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:56:55 +03:00
Aliaksandr Valialkin	10c63f5853	app/vmselect: `make fmt` after `f3ece83e67`	2022-07-05 14:56:53 +03:00
Aliaksandr Valialkin	8516670582	app/vmselect/promql: properly calculate histogram_quantile over unexpected `le` buckets See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819	2022-07-05 13:20:41 +03:00
Artem Navoiev	82581ab728	Docs: Operator Additional Scrape Configuration - update docs (#2826 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-07-05 13:20:41 +03:00
Roman Khavronenko	018782c24e	dashboards: small visual tweaks for vmagent's dashboard (#2828 ) * remove lines filling * filter series with zero values * update descriptions Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-05 13:20:41 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin	757f4fd9f9	vendor: `make vendor-update`	2022-07-04 12:01:34 +03:00
Aliaksandr Valialkin	24fc96dc1f	docs/Troubleshooting.md: add a link to `Monitoring` chapter added at `41d1834a99`	2022-07-04 11:57:14 +03:00
Roman Khavronenko	51209c114a	docs: update Troubleshooting guide (#2809 ) Follow-up after `a4d9388ecb` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-04 11:57:12 +03:00
Roman Khavronenko	78da4b5f5e	docs: warn about potential issue with read queries for 1.78.0 (#2818 ) * docs: warn about potential issue with read queries for 1.78.0 Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: warn about potential issue with read queries for 1.78.0 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-04 11:47:41 +03:00
Dmytro Kozlov	7389fc9adf	vmui: disable ripple on nested component, enable copy text on button (#2808 )	2022-07-04 11:47:39 +03:00
Aliaksandr Valialkin	f3b905dde1	docs/Cluster-VictoriaMetrics.md: mention about -storage.maxDailySeries and -storage.maxHourlySeries options in `resource usage limits` chapter	2022-06-30 23:17:14 +03:00
Aliaksandr Valialkin	fa08220d27	app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787	2022-06-30 22:40:26 +03:00
Aliaksandr Valialkin	1e6b0a1f54	app/vmagent/remotewrite: do not shadow `headers` global variable in getAuthConfig	2022-06-30 20:18:43 +03:00
Aliaksandr Valialkin	0f6c17ed06	app/vmagent/remotewrite: clarify descriptions for `-remoteWrite.*` options, which must be set per each `-remoteWrite.url`	2022-06-30 20:18:42 +03:00
Aliaksandr Valialkin	7fc03a1deb	app/vmagent/remotewrite: add `-remoteWrite.header` command-line flag for setting additional http headers to send to -remoteWrite.url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin	4fb0f15322	all: readability improvements for query traces - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:19:43 +03:00
Aliaksandr Valialkin	ffd88d0b76	vendor: update github.com/VictoriaMetrics/metricsql from v0.44.0 to v0.44.1	2022-06-30 17:37:57 +03:00
Dmytro Kozlov	67753cc6f0	vmui: update render logic for nested component (#2795 ) * vmui: update render logic for nested component, avoid rerender, remove local storage usage for tracing flag * docs/url-examples.md: fix various documentation issues there * docs: add Troubleshooting doc This doc contains troubleshooting guides for typical problems with VictoriaMetrics. * docs/Troubleshooting.md: add troubleshooting guide for cluster instability * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-30 15:48:22 +03:00
Aliaksandr Valialkin	2a70a9296e	docs/Troubleshooting.md: refer to subqueries doc, since this is the most frequent source of improperly consutructed queries	2022-06-30 15:10:02 +03:00
Aliaksandr Valialkin	edf39f9281	docs/Troubleshooting.md: refer to capacity planning docs	2022-06-30 15:00:21 +03:00
Aliaksandr Valialkin	47f3c4b12b	docs/Troubleshooting.md: various typo fixes and clarifications	2022-06-30 14:57:05 +03:00
Aliaksandr Valialkin	7c54cc2123	docs/Troubleshooting.md: formatting fixes	2022-06-30 14:39:15 +03:00
Aliaksandr Valialkin	74338ef300	docs/Troubleshooting.md: add troubleshooting guide for cluster instability	2022-06-30 14:36:18 +03:00
Aliaksandr Valialkin	da3179667e	docs/CHANGELOG.md: document `bdf9f4669a`	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	92e6c55b75	docs: add Troubleshooting doc This doc contains troubleshooting guides for typical problems with VictoriaMetrics.	2022-06-30 13:52:44 +03:00
ttyv	00956e585d	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786 ) Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	0b76d07c21	docs/url-examples.md: fix various documentation issues there	2022-06-29 11:58:34 +03:00
Aliaksandr Valialkin	daefb64f38	app/vmselect: expose additional histograms at `/metrics` page, which may help get more insights for the query workload This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792	2022-06-28 20:18:31 +03:00
Aliaksandr Valialkin	9adff0b686	app/vmselect/promql: show the number of scanned samples in the query trace	2022-06-28 19:32:03 +03:00
Aliaksandr Valialkin	c21f5a508f	app/vmselect/prometheus: reduce the default value for -search.maxSeries from 100k to 30k Production experience shows that 100k is too big for /api/v1/series . It leads to increased CPU usage when Grafana queries /api/v1/series over VictoriaMetrics with big number of time series during auto-completion and when modifying template variables.	2022-06-28 18:22:24 +03:00
Aliaksandr Valialkin	5e22e6046d	docs/Single-server-VictoriaMetrics.md: mention about -search.maxTagValueSuffixesPerSearch command-line flag in `resource limits` docs	2022-06-28 18:17:49 +03:00
Aliaksandr Valialkin	7d5d33fd71	lib/storage: return marshaled metric names from SearchMetricNames Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin	69bbdf7304	vendor: `make vendor-update`	2022-06-28 14:55:41 +03:00
Aliaksandr Valialkin	15da802f5f	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:52:39 +03:00
Aliaksandr Valialkin	36edb1912b	app/vmstorage: rename "transport" package to "servers" package for better clarity	2022-06-28 14:04:14 +03:00
Aliaksandr Valialkin	399d4c36ae	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin	64505e924d	app/vmstorage: extract vmselect api server into a separate package - lib/vmselectapi This opens doors for implementing vmselect api server at vmselect level, so top-level vmselect could query lower-level vmselect nodes in the same way as it queries vmstorage nodes. This will create the ability to create highly available querying architecture when multiple independent VictoriaMetrics clusters with the same data are located in distinct availability zones. In this case we can use top-level vmselect instead of Promxy for simultaneous querying of all the clusters in all the AZs.	2022-06-27 14:20:41 +03:00
Aliaksandr Valialkin	1af6efd737	docs/Single-server-VictoriaMetrics.md: mention about -search.maxTagValueSuffixesPerSearch command-line flag in `resource limits` docs	2022-06-27 14:02:01 +03:00
Aliaksandr Valialkin	a667d339be	app/vmselect/netstorage/netstorage.go: group metrics in order to improve readability a bit	2022-06-27 14:00:24 +03:00
Roman Khavronenko	50c0eb4c4e	vmalert: make `__name__` available for templating in alerts (#2783 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-27 13:53:55 +03:00
Aliaksandr Valialkin	a38568ddfe	docs/CHANGELOG.md: document `45f20ad1aa`	2022-06-27 13:53:26 +03:00
Aliaksandr Valialkin	6386f117c8	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin	4b41a05ca7	app/vmalert: load static js and css from proper paths if `-http.pathPrefix` command-line flag is set This is a follow-up for `b104f67beb`	2022-06-27 13:12:57 +03:00
Aliaksandr Valialkin	926fccbb8d	lib/storage: add querytracer to more contexts querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 12:53:49 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	08de733924	app/vmselect/netstorage: assume the response is full if up to -replicationFactor-1 vmstorage nodes are unavailable This is a follow-up for `ee5c502446` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767	2022-06-27 12:21:26 +03:00
Roman Khavronenko	572db17857	vmalert: use absolute path for assets (#2784 ) Using relative path breaks assets loading on alert view page. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-27 00:47:36 +03:00
Aliaksandr Valialkin	bc9d704ef4	app/vmselect/netstorage: remove Get prefix from netstorage functions This makes these function names more consistent with the server side	2022-06-27 00:37:49 +03:00
Aliaksandr Valialkin	ea2aa617e5	docs/CHANGELOG.md: update after `e40d015e9a`	2022-06-24 18:03:56 +03:00
Aliaksandr Valialkin	94b81165d0	all: limit the maximum memory usage for regexp cache, which stores parsed regular expressions in MetricsQL queries Previously the cache could store 10K unique regexps. When every regexp is huge (e.g. hundreds of kilobytes), then the total cache size could grow to multiples of gigabytes. Now the cache size is limited by the total length of all cached regexps. So huge regexps won't result in high memory usage for the cache.	2022-06-24 17:58:47 +03:00
hagen1778	e40d015e9a	vmselect: make `vm_partial_results_total` consistent Metrics `vm_partial_results_total` and `vm_requests_total` serving the similar purpose, but contain inconsistent set of labels. This change updates `vm_partial_results_total` labels to be consistent with `vm_requests_total`. The change breaks backward compatibility with assumption that `vm_partial_results_total` wasn't widely used, since it is not documented and absent in the alerts and dashboards. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-24 13:50:26 +02:00
Dmytro Kozlov	f28cbcc7b5	vmui: added query tracing (#2748 ) * vmui: added query tracing * vmui: updated ui * vmui: update tracing logic, fix bugs, disable tracing by default * vmui: use empty message as props * vmui: fixed ui, added delete for each tacing data, show query in header * vmui: added timelines * vmui: speedup render * vmui: use memo for sorting * vmui: use Trace model, remove unused functions, simplify part of code * vmui: update recursive logic * vmui: fix set query to header * vmui: code cleanup, remove unused code * vmui: remove unused type, rename component * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-23 23:00:45 +03:00
Nikolay	ee5c502446	app/vmselect: fixes partial response with replicationFactor (#2777 ) * app/vmselect: fixes partial response with replicationFactor Allow partial response if it meets replicationFactor configured at vmselect https://t.me/VictoriaMetrics_ru1/38490 * docs/CHANGELOG.md: document this change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-23 20:17:24 +03:00
Yurii Kravets	f0c1edb175	docs: Update CHANGELOG Update notes (#2776 ) * docs: Update CHANGELOG Update notes Specified the reason why `vmselect` and `vmstorage` nodes may log communication errors.	2022-06-23 19:40:20 +03:00
Aliaksandr Valialkin	e0ce6c0ff8	app/vmstorage/transport: refactoring: split Server into VMInsertServer and VMStorageServer This makes the code more clear	2022-06-23 19:20:09 +03:00
Aliaksandr Valialkin	71b0dfdefa	lib/promscrape: always send stale markers with the real scrape timestamp This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:49:13 +03:00
Roman Khavronenko	fc03950efa	dashboards: update cluster dashboard (#2773 ) * dashboards: update cluster dashboard * add assisted merges panel https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2754 * add mem panel per each component * remove lines filling for some panels for clarity Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update dashboards/victoriametrics.json	2022-06-23 09:46:28 +02:00
Denys Holius	d7c4b01472	Adds a list of supported architectures (#2769 ) * add list of supported architectures * Update docs/BestPractices.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-06-22 22:01:29 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	7429dfbe9e	app/vmselect: add `-search.setLookbackToStep` command-line flag for making the gap filling algorithm similar to InfluxDB data model This option should override `-search.maxStalenessInterval` for most cases when users migrate from InfluxDB to VictoriaMetrics	2022-06-22 14:20:02 +03:00
Aliaksandr Valialkin	3e0f364489	docs/Cluster-VictoriaMetrics.md: small fixes	2022-06-22 13:42:07 +03:00
Aliaksandr Valialkin	4f64da874d	app/vmselect: typo fix in the exported metric name: vm_http_request_total -> vm_http_requests_total	2022-06-22 13:18:03 +03:00
Roman Khavronenko	54f0f2d384	docs: follow-up for `197d3cdd74` (#2766 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-22 13:18:03 +03:00
云原生驿站	67e5833ced	docs: supplement vmalert downsampling docs (#2765 ) Co-authored-by: 吴典秋 <muti_kube@163.com>	2022-06-22 13:18:03 +03:00
Aliaksandr Valialkin	dceca7e864	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package This is a follow-up for `fe2269b999`	2022-06-21 20:27:30 +03:00
Aliaksandr Valialkin	fe2269b999	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package This package already has the same name, so there is no need in explicit name	2022-06-21 20:24:28 +03:00
Denys Holius	432b261c13	url-examples: added curl output after deleting metrics (#2764 ) docs: add more details to url-examples for series deleting	2022-06-21 17:56:00 +03:00
Loki's Wager	ca4730c00f	BugFix part_header.go (#2763 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin	9362da2e7f	docs: update `-help` output for vmbackup, vmbackupmanager, vmgateway and vmrestore components	2022-06-21 15:49:26 +03:00
Aliaksandr Valialkin	597bce4f55	docs: update docs after `e4d6b750f6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753	2022-06-21 14:01:25 +03:00
Aliaksandr Valialkin	288d13af8d	lib/netutil: parallelize background pings for remote addresses This should improve the time needed for determining unavailale remote addresses across big numer of ConnPool's. This is a follow-up for `a1629bd3be` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-21 13:32:27 +03:00
Aliaksandr Valialkin	24097f2417	docs/CHANGELOG.md: cut v1.78.0	2022-06-20 18:11:04 +03:00
Yurii Kravets	14397ba23e	Changed the level type in alerts.yml for TooManyLogs alert (#2759 ) alerts: filter out non error log messages for `TooManyLogs` Info and Warn error levels aren't always a result of malfunctioning or faulty state. So we filter them out.	2022-06-20 16:45:52 +02:00
Aliaksandr Valialkin	a1629bd3be	lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped until the background goroutine determines that the given address can be successfully dialed. This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711 This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756 The main differences are: - The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool. This makes possible re-using this feature everywhere lib/netutil.ConnPool is used. - The check doesn't take into account handshake errors for already established connections. Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.	2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin	45e9732764	docs: follow-up after `e4d6b750f6`	2022-06-20 17:15:52 +03:00
Nikolay	15662c0f29	lib/httpserver: adds flagsAuthKey command-line flag (#2758 ) * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:15:51 +03:00
Aliaksandr Valialkin	b28c6febf9	app/{vminsert,vmselect}: add `-vmstorageDialTimeout` command-line flag for tuning the maximum time needed for establishing connections to vmstorage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-20 15:17:34 +03:00
Aliaksandr Valialkin	0abf46d66a	docs/Articles.md: add a link to https://www.sobyte.net/post/2022-05/victoriametrics-bloomfilter/	2022-06-20 14:44:11 +03:00
Aliaksandr Valialkin	032d4fdf7d	vendor: `make vendor-update`	2022-06-20 14:31:57 +03:00
Aliaksandr Valialkin	079fdd3158	all: update Go builder for production builds from 1.18.2 to 1.18.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.18.3+label%3ACherryPickApproved	2022-06-20 14:30:48 +03:00
Aliaksandr Valialkin	b2cfb8faf7	app/vmstorage/transport: call vmselectRequestCtx.readSearchQuery() in processVMSelectDeleteMetrics Previously the processVMSelectDeleteMetrics was calling separate functions from readSearchQuery(). It is better from readability and maintenance PoV to substitute it with readSearchQuery call.	2022-06-20 14:23:17 +03:00
Aliaksandr Valialkin	270ad39359	lib/storage: properly take into account already registered series when `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are enabled The commit `5fb45173ae` takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:53:41 +03:00
Roman Khavronenko	3ada676879	docs: reference links from key concepts (#2745 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-19 23:14:30 +03:00
Aliaksandr Valialkin	fe9f59fcd6	all: replace `bash` with `console` blocks in all the *.md files This is a follow-up for `954a7a6fc6`	2022-06-19 23:02:02 +03:00
Artem Navoiev	42c69ae74e	docs: replace bash code block type with console (#2746 )	2022-06-19 23:02:00 +03:00
Aliaksandr Valialkin	4ae1c5655f	docs/CHANGELOG.md: document `ef7f52e0e6`	2022-06-19 22:49:11 +03:00
Roman Khavronenko	3e45e1ff63	Vmalert notifiers (#2744 ) * vmalert: remove head of line blocking for sending alerts This change makes sending alerts to notifiers concurrent instead of sequential. This eliminates head of line blocking, where first faulty notifier address prevents the rest of notifiers from receiving notifications. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: make default timeout for sending alerts 10s Previous value of 1m was too high and was inconsistent with default timeout defined for notifiers via configuration file. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: linter checks fix Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-19 22:49:10 +03:00
Aliaksandr Valialkin	7a79e7c0ef	lib/storage: create per-day indexes together with global indexes when registering new time series Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:32:41 +03:00
Aliaksandr Valialkin	88e1221b35	lib/storage: do not register new series if `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are exceeded Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:03:02 +03:00
Aliaksandr Valialkin	c5ac176153	lib/storage: reset metric id caches for the previous and the current hour Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:02:51 +03:00
Aliaksandr Valialkin	149c5c9381	docs: update docs on how to add tags to metrics collected by DataDog agent Follow-up for `f16072c3c1`	2022-06-17 13:12:21 +03:00
Dmytro Kozlov	5bc13e2fe8	vmui: added focusLabel, enable cardinality app configuratior (#2736 ) * vmui: added focusLabel, enable app configuratior * vmui: set focusLabel if {labelName!=""} * wip * docs/CHANGELOG.md: mention about focusLabel feature in cardinality explorer Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2730 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-17 13:04:09 +03:00
Dmytro Kozlov	82440e76c5	doc: added workaround for datadog agent (#2712 ) * Added workaround for datadog agent * docs: update datadog workaround * doc: update doc description * Apply suggestions from code review * docs: `make docs-sync` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-16 22:24:17 +03:00
Roman Khavronenko	287bb96758	docs: mention sandbox update in release procedure (#2724 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-16 20:51:41 +03:00
Roman Khavronenko	cbe39bbb8d	vmselect: limit `end` param max value by 2d in future (#2729 ) * vmselect: limit `end` param max value by 2d in future The change is applied only to service handlers like `/labels` or `/series` and limits the `end` param by max value <= now() + 2 days. The same limit is applied for the ingested data, so no reason to allow to request data in future far than that. The change is also needed for corner cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2669 where too high `end` value triggers inefficient global index search. Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-16 20:51:40 +03:00
Aliaksandr Valialkin	cda3ce09e5	docs/vmagent.md: typo fix: configued -> configured	2022-06-16 20:51:40 +03:00
Roman Khavronenko	246d2df361	dashboards: add cpu usage panels per each component type (#2723 ) The change adds extra panel per each component, showing the amount of used CPU cores and the limit (summary of all instances). https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2696 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-16 20:49:55 +03:00
Aliaksandr Valialkin	450aa0ae5a	lib/promrelabel: support `action: graphite` relabeling Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:25:49 +03:00
Roman Khavronenko	ba7ece02c4	docs: add multiple-remote-writes topology to vmalert (#2738 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-16 20:21:12 +03:00
Aliaksandr Valialkin	da1d1e83df	app/{vmselect,vmstorage}: properly pass seriesCountByLabelName and seriesCountByFocusLabelValue entries from vmstorage to vmselect	2022-06-16 10:44:29 +03:00
Aliaksandr Valialkin	b4e75a0b89	docs/CHANGELOG.md: document dd327bfa9e2c69fe21ab1d92c14636733d7c5620	2022-06-15 18:41:26 +03:00
Aliaksandr Valialkin	14ce7b0e25	docs/CHANGELOG.md: document 00719e5779a3e4eeedb74cb3d25a9ecfe0e16063	2022-06-15 18:09:23 +03:00
Aliaksandr Valialkin	ee9954082f	app/vmselect/netstorage: properly aggregate seriesCountByLabelName and seriesCountByFocusLabelValue obtained from multiple vmselect nodes at /api/v1/status/tsdb	2022-06-15 16:48:40 +03:00
Aliaksandr Valialkin	45fa9d798d	app/vmselect: accept `focusLabel` query arg at /api/v1/status/tsdb	2022-06-14 18:39:00 +03:00
Aliaksandr Valialkin	fb77843639	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:34:13 +03:00
Dmytro Kozlov	ed9b4c6f6d	vmui: refactor Cardinality panel (#2726 ) * vmui: refactor Cardinality panel * vmui: change width of the search panel * vmui: code cleanup * vmui: code cleanup * vmui: fixed vulnerability (npm audit fix) * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-14 14:40:44 +03:00
Aliaksandr Valialkin	ce953d5e95	docs: update command-line flags' descriptions according to recent changes	2022-06-14 13:28:23 +03:00
Aliaksandr Valialkin	3167fbc21d	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:28:21 +03:00
Nikolay	e23af8f05c	lib/httpserver: backport changes from master branch (#2697 ) * lib/httpserver: backport changes from master branch adds basicAuth adds authKey check for /metrics and /debug/pprof requests it should improve security for cluster components * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-14 13:02:44 +03:00
Roman Khavronenko	0065b13243	docs: reduce free disk recommendation from 30% to 20% (#2728 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-14 12:55:41 +03:00
Aliaksandr Valialkin	12618925fc	docs/guides/migrate-from-influx.md: suggest more real-world value for `-search.maxStalenessInterval` Suggest `-search.maxStalenessInterval=10s` instead of `-search.maxStalenessInterval=1ms`, since `1ms` would result in empty graphs in most cases, since the interval between data points on the graph is usually much higher than 1ms. For example, if the graph shows time range of one hour and it contains 1000 points, then the interval between points on the graph would equal to 3600s/1000=3.6 seconds.	2022-06-14 12:51:35 +03:00
Roman Khavronenko	9873001539	docs: migrating from influx (#2720 ) Mention `-search.maxStalenessInterval` flag and its effect on query engine. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-14 12:51:33 +03:00
Aliaksandr Valialkin	b48ca34e05	docs/Single-server-VictoriaMetrics.md: recommend running all the VictoriaMetrics components behind auth proxy in Security chapter	2022-06-13 10:30:52 +03:00
Aliaksandr Valialkin	e1fc4db109	vendor: `make vendor-update`	2022-06-13 10:07:19 +03:00
Aliaksandr Valialkin	c846d0db01	docs/CHANGELOG.md: document `99dbe7f9d4`	2022-06-13 10:07:19 +03:00
Wataru Manji	0cd750fa5e	Add remote-write headers (#2701 ) Co-authored-by: Wataru Manji <wataru.manji@linecorp.com>	2022-06-13 10:07:19 +03:00
Aliaksandr Valialkin	8128c6db6b	app/vmagent: follow-up after `4583ed23a8`	2022-06-13 09:56:27 +03:00
Dmytro Kozlov	f0cb3c4999	Added a stub for datadog endpoint (#2710 ) * Added a stub for datadog endpoint * Update app/vmagent/main.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-13 09:56:26 +03:00
Roman Khavronenko	81b6efea9e	Readme cleanup (#2715 ) * docs: minor styling and wording changes Changes made after reading https://developers.google.com/tech-writing Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: set proper types for code blocks Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: add `copy` wrapper for some commands Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: sync Signed-off-by: hagen1778 <roman@victoriametrics.com> * docs: resolve conflicts Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-13 09:49:59 +03:00
Yury Molodov	65107a4585	vmui: enhancements (#2638 ) (#2717 ) * feat: make datepicker to be set to last 30 min by default * fix: correct spinner while loading data * feat: change legend style * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-13 09:49:58 +03:00
Yury Molodov	7a3b377ac0	vmui: fix relative time and query params (#2716 ) * fix: correct update query params * fix: change select relative time	2022-06-13 09:34:06 +03:00
Aliaksandr Valialkin	9d8c37d280	docs/CHANGELOG.md: refer to the issue, which should be solved after the optimization to /api/v1/labels and /api/v1/label/.../values is added The optimization has been added in `374beb350e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533	2022-06-12 14:32:57 +03:00
Aliaksandr Valialkin	4af43a4a75	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:28:37 +03:00
Aliaksandr Valialkin	61e03f172b	app/vmselect: optimize `/api/v1/labels` and `/api/v1/label/.../values` handlers when `match[]` query arg is passed to them	2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin	4a94cd81ce	app/vmselect: add optional `limit` query arg to `/api/v1/labels` and `/api/v1/label_values` endpoints This arg allows limiting the number of sample values returned from these APIs	2022-06-10 10:24:07 +03:00
Aliaksandr Valialkin	1335698ba7	app/vmselect/prometheus: extract common code for obtaining common query args into getCommonParams() function	2022-06-09 20:33:36 +03:00
Aliaksandr Valialkin	cb39eada77	all: improve query tracing coverage for indexdb search Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:04:02 +03:00
Aliaksandr Valialkin	ad44eadd94	app/vmselect/graphite: remove additional redundant Request.ParseForm() calls after `38c785b851`	2022-06-09 13:22:23 +03:00
Artem Navoiev	5b4b922433	dashboards: update cluster by tenant dashboard (#2695 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-06-09 13:15:40 +03:00
Roman Khavronenko	3c09d25039	vmalert: followup for `76f05f8670` (#2706 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-09 13:15:35 +03:00
Howie	4afd7aa695	feat: rule limit (#2676 ) vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 13:15:33 +03:00
Aliaksandr Valialkin	7fe60dad10	app/vmselect: remove redundant calls to Request.ParseForm() Request.ParseForm() is implicitly called by the first call to Request.FormValue()	2022-06-09 13:09:52 +03:00
Aliaksandr Valialkin	a9ea3fee38	lib/querytracer: make it easier to use by passing trace context message to New and NewChild The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:16:12 +03:00
Aliaksandr Valialkin	5524d13a03	docs/Single-server-VictoriaMetrics.md: explain why free disk space shortage may negatively impact VictoriaMetrics performance	2022-06-08 20:04:30 +03:00
Aliaksandr Valialkin	2520b8efb8	docs: add a link to cardinality explorer playground	2022-06-08 19:40:12 +03:00
Aliaksandr Valialkin	2b343d8bd0	app: properly collect and merge /api/v1/status/tsdb info from vmstorage nodes The collection has been broken in `f2754c3e90` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233	2022-06-08 19:26:09 +03:00
Aliaksandr Valialkin	8d98b8ae10	docs: refer to cardinality explorer Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233	2022-06-08 19:18:31 +03:00
Dmytro Kozlov	f2754c3e90	Cardinality explorer (#2625 ) * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:54:27 +03:00
Aliaksandr Valialkin	ba406ff28c	docs/CHANGELOG.md: document `63b538ecd1`	2022-06-07 15:53:08 +03:00
Roman Khavronenko	2b5e1dee91	vmagent: update SD duration histogram metric if SD is active (#2677 ) The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:53:06 +03:00
Aliaksandr Valialkin	d830f43608	app/vminsert/main.go: add missing `strings` import after `0c5e199469`	2022-06-07 15:42:23 +03:00
Aliaksandr Valialkin	c07f99cd4f	docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2685	2022-06-07 15:39:53 +03:00
Wataru Manji	64f7095c3a	add Content-Encoding Header (#2685 ) Co-authored-by: Wataru Manji <wataru.manji@linecorp.com>	2022-06-07 15:34:25 +03:00
Aliaksandr Valialkin	c7ff4505d2	docs/CHANGELOG.md: document backwards-incompatible changes in communication protocol between vmselect and vmstorage The changes are related to the added query tracing in `afced37c0b` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-07 15:25:58 +03:00
Aliaksandr Valialkin	2b72d21840	docs/CHANGELOG.md: document `e755d0ec3f`	2022-06-07 15:21:10 +03:00
elProxy	0c5e199469	Support legacy datadog agent (#2670 ) dd-agent v5 can issue some requests with trailing slashes. (e.g. `526559be73/ddagent.py (L303)`) Trim trailing slashes for request on /datadog/ paths to accomodate for that. Co-authored-by: Pierre Rossi <pierre.rossi@schibsted.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-07 15:21:09 +03:00
Aliaksandr Valialkin	0b4f767c53	docs/CHANGELOG.md: document `1ee1e986da`	2022-06-07 15:02:57 +03:00
Roman Khavronenko	5f33445f66	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673 ) Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:02:55 +03:00
Aliaksandr Valialkin	42d3a19190	README.md: sync the change with docs/Cluster-VictoriaMetrics.md after `50393d0024`	2022-06-07 14:53:01 +03:00
Aliaksandr Valialkin	33afaa49d9	docs: run `make docs-sync` after `fda8da297e`	2022-06-07 14:51:32 +03:00
Dmytro Kozlov	50393d0024	docs: fixed typos (#2680 ) * docs: fixed typos * Update README.md * Update docs/README.md * Update docs/Single-server-VictoriaMetrics.md * docs: added examples with start and end params in request * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-07 14:51:31 +03:00
Aliaksandr Valialkin	933f4ae148	docs/Cluster-VictoriaMetrics.md: run `make docs-sync` after `1c96dce367`	2022-06-07 14:30:09 +03:00
Luckz	1c96dce367	README.md: rephrase a few things for grammar and clarity (#2687 ) Signed-off-by: Luckz <224748+Luckz@users.noreply.github.com>	2022-06-07 14:29:05 +03:00
Aliaksandr Valialkin	b4909351c5	docs: run `make docs-sync` after `a439e887a3`	2022-06-07 14:27:37 +03:00
Aliaksandr Valialkin	b6e3c12811	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin	68b6ddfb14	all: follow-up after `8edb390e21` - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin	3dbb19d624	lib/promscrape/discovery/kubernetes: follow-up after `006b8c7534` - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin	38342f959a	vendor: `make vendor-update`	2022-06-06 13:25:11 +03:00
Aliaksandr Valialkin	6d2832f413	docs/Articles.md: add a link to https://percona.community/blog/2022/06/02/long-time-keeping-metrics-victoriametrics/	2022-06-06 13:15:42 +03:00
Aliaksandr Valialkin	dd0d773c13	lib/promauth: follow-up after `006b8c7534` - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	6c2fb9d8c4	lib/promscrape: add `-promscrape.cluster.name` command-line flag This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 01:11:23 +03:00
Dmytro Kozlov	ce8aade80e	lib/promscrape: adds service discovery visualization for /targets page(#2675 ) * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-04 01:11:23 +03:00
Nikolay	72e43ef2fe	lib/promscrape/discovery/kubernetes: follow-up after `0b5c874911` (#2672 )	2022-06-04 01:11:23 +03:00
hadesy	28d4624f60	promscrape/discovery: support kubeconfig (#2533 )	2022-06-04 01:11:23 +03:00
Roman Khavronenko	d956f6f68e	Dashboar cluster update (#2674 ) * dashboard: fix query for `CPU percentage` panel Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboard: replace Uptime panel with Version panel Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-02 16:03:48 +02:00
Aliaksandr Valialkin	70bb84739d	docs/Single-server-VictoriaMetrics.md: small clarification about custom cache size tuning	2022-06-01 14:57:39 +03:00
Aliaksandr Valialkin	cc226e6ebe	docs/CHANGELOG.md: follow-up after `2177089f94`	2022-06-01 14:57:39 +03:00
Roman Khavronenko	e9ee043879	lib/storage: make `indexdb/tagFilters` cache size configurable (#2667 ) The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Roman Khavronenko	bca90d7148	promrelabel: add support of `lowercase` and `uppercase` relabeling actions (#2665 ) * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Aliaksandr Valialkin	c92bc5394f	app/vmselect/netstorage: properly read trace from vmstorage when it returns error message to vmselect	2022-06-01 14:35:00 +03:00
Aliaksandr Valialkin	fedfc9e686	lib/storage: stop background merge when storage enters read-only mode This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin	f133756f02	deployment/docker: update base image from alpine:3.15.4 to alpine:3.16.0	2022-06-01 02:54:53 +03:00
Aliaksandr Valialkin	afced37c0b	all: add initial support for query tracing See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin	386f6110ec	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2022-06-01 01:43:25 +03:00
Dima Lazerka	264b98192e	Fix nth-check version for css-select dep (#2666 ) Fixes security vulnerability in nth-check version <=1.0.2 My previous version pin was insufficient, as it was imported again through a different (svgo -> css-select).	2022-06-01 01:43:24 +03:00
Aliaksandr Valialkin	af4daede2e	vendor: `make vendor-update`	2022-05-31 12:58:28 +03:00
Aliaksandr Valialkin	945e9fa8c4	lib/storage: `make fmt`	2022-05-31 12:42:48 +03:00
Aliaksandr Valialkin	e83b96366f	docs/CHANGELOG.md: follow-up after `11f91532c5` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Aliaksandr Valialkin	727cc119b6	lib/storage: do not take into account series from the next day when `match[]` filter is passed to /api/v1/status/tsdb	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	cd1fa2e4cd	issue-2594: use embedded for static files (#2650 ) embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Howie	09fed19ba5	chore: remove duplicated code (#2657 ) Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-05-30 12:27:47 +03:00
Howie	9c5f998438	fix: docs (#2658 ) Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-05-30 12:27:19 +03:00
Roman Khavronenko	4f84afb032	docs: add more details to the key concepts (#2648 ) - data modification - more applications for gauges - recommendations for instrumenting app with metrics Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-30 12:26:37 +03:00
Dmytro Kozlov	6add79143b	removed redundant return (fixed linter) (#2647 ) * removed redundant return * updated lint package version	2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin	0617c5eaca	README.md: sync with docs/Cluster-VictoriaMetrics.md after the commit `3299865620`	2022-05-30 12:21:42 +03:00
Roman Khavronenko	46c06334ee	dashboards: use `vm_concurrent_select_current` instead of `vm_concurrent_queries` (#2655 ) Using metric `vm_concurrent_queries` in relation to `vm_concurrent_select_capacity` is incorrect. Switching to `vm_concurrent_select_current` in `Concurrent selects` panel. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-30 12:16:24 +03:00
Dima Lazerka	3299865620	Change img urls to the same domain (#2649 )	2022-05-27 20:13:49 +02:00
Aliaksandr Valialkin	f0ab5ca61f	app/vmselect/vmui: `make vmui-update` after `492a615a88`	2022-05-26 09:43:30 +03:00
Yury Molodov	b9ef9ddbfd	vmui: import dashboards (#2642 ) * fix: switch dashboards import to fetch * make vmui-update	2022-05-26 09:43:29 +03:00
Aliaksandr Valialkin	f149d56ac2	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 23:00:30 +03:00
Roman Khavronenko	d367a65707	docs: update single version docs by adding extra links and formatting (#2643 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-25 23:00:28 +03:00
Aliaksandr Valialkin	5924511960	vendor: `make vendor-update`	2022-05-25 21:52:54 +03:00
Aliaksandr Valialkin	38beb9fe04	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag This is a follow-up for `0fbf59199a` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:07:14 +03:00
阳明	e4df648ea0	lib/storage: Remove the effect of time zone on next retention period (#2568 ) (#2574 )	2022-05-25 15:10:19 +03:00
Aliaksandr Valialkin	05e47482a5	docs/CHANGELOG.md: document `9e343faa41`	2022-05-25 15:05:42 +03:00
Roman Khavronenko	cd8e58d0ae	docs: mention guide for migrating from inlfux in vmctl readme (#2640 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-25 15:05:12 +03:00
spectvtor	c5df5c9a95	fix alert relabeling (#2633 )	2022-05-25 15:05:10 +03:00
Aliaksandr Valialkin	df9ffa6854	app: expose `/api/v1/status/config` endpoint in the same way as Prometheus does This endpoint is needed for third-party tools. See https://prometheus.io/docs/prometheus/latest/querying/api/#config	2022-05-25 10:00:00 +03:00
Nikolay	514843be77	dashboards: adds dashboard for operator (#2621 ) Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Adds proper interval to rate functions	2022-05-23 11:49:03 +03:00
Aliaksandr Valialkin	fe631850ee	deployment/docker: update Go builder from 1.18.1 to 1.18.2 See https://github.com/golang/go/issues?q=milestone%3AGo1.18.2+label%3ACherryPickApproved	2022-05-23 10:58:38 +03:00
Aliaksandr Valialkin	090c331c12	docs/Single-server-VictoriaMetrics.md: mention `Cardinality limiter` docs at `Resource usage limits` section	2022-05-23 10:55:42 +03:00
Nikolay	754d3b1f93	Updates operator docs (#2622 ) * docs/operator: adds information about VMAgent statefulMode * docs/operator: adds description for alertmanager configuration * docs/operator: adds description for configuration syncronization https://github.com/VictoriaMetrics/operator/issues/124 Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-05-23 10:55:42 +03:00
Roman Khavronenko	d597cdd06f	vmalert: mention how to build a custom image (#2626 ) Thanks to @f41gh7 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-23 10:55:42 +03:00
Aliaksandr Valialkin	513f9537a9	docs/CHANGELOG.md: cut v1.77.2	2022-05-21 02:26:29 +03:00
Aliaksandr Valialkin	cdbee6f470	docs/CHANGELOG.md: group vmalert features together	2022-05-21 01:52:13 +03:00
Aliaksandr Valialkin	35ed63f32e	docs/CHANGELOG.md: document `2cf586da78`	2022-05-21 01:17:49 +03:00
Roman Khavronenko	7406665fc3	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615 ) * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:17:21 +03:00
Roman Khavronenko	88c4c6f465	vmalert: add new metric `vmalert_iteration_interval_seconds` (#2623 ) The new metric shows the configured evaluation interval per group. Metric updates its value when group's interval is changed during hot reload. The new metric can be used to estimate how close group is to start missing evaluation rounds. The following query will show the % of used time by the group to evaluate all rules before the next round: ``` (max(vmalert_iteration_duration_seconds{quantile="0.99"}) / vmalert_iteration_interval_seconds) * 100 ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2618 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-21 01:13:01 +03:00
Yurii Kravets	a723de6ae1	Update README.md (#2624 )	2022-05-21 01:07:04 +03:00
Aliaksandr Valialkin	aba00b8cb2	docs: update the description for command-line flags according to recent changes	2022-05-20 15:11:37 +03:00
Aliaksandr Valialkin	4409b9f376	app/vmselect/vmui: `make vmui-update`	2022-05-20 14:56:25 +03:00
Aliaksandr Valialkin	e0bd035467	vendor: `make vendor-update`	2022-05-20 14:48:23 +03:00
Aliaksandr Valialkin	d49dbec662	Makefile: explicitly specify go1.17 compatibility when running `go mod tidy` at `make vendor-update` This is needed because go1.17 is the minimum supported version of Go, which is needed for building VictoriaMetrics	2022-05-20 14:42:06 +03:00
Aliaksandr Valialkin	4ca5d97459	docs/CHANGELOG.md: document `3df8caca15`	2022-05-20 14:24:36 +03:00
Roman Khavronenko	e9165b041e	docs: add migration guide for influxdb (#2619 ) docs: add migration guide for influxdb Co-authored by @denisgolius Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:24:35 +03:00
Aliaksandr Valialkin	386ef69124	docs/CHANGELOG.md: formatting fixes	2022-05-20 14:22:02 +03:00
Aliaksandr Valialkin	98123e95dd	docs/CHANGELOG.md: formatting fix for the issue url https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2607	2022-05-20 14:22:02 +03:00
Aliaksandr Valialkin	c5765dd6ad	docs/CHANGELOG.md: link to the feature request about reusable templates in vmalert Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2510	2022-05-20 14:21:35 +03:00
Roman Khavronenko	f71d15c79e	docs: follow-up after `4b3eb40658` (#2602 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:21:09 +03:00
Roman Khavronenko	dcd8387601	docs: fix a typo of mentioning scrape support for vmselect (#2617 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:20:37 +03:00
Boris Petersen	3a8b4fab97	Add ability to sign requests for all AWS services (#2604 ) This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-20 14:20:00 +03:00
Roman Khavronenko	d814c83b21	vmalert: remove a line added for debug (#2611 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:08:57 +03:00
Roman Khavronenko	2aeb00f98f	vmalert: support `scalar` type in response (#2610 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2607 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:08:19 +03:00
Roman Khavronenko	a07ddf9b65	vmalert: support strings in `humanize.*` templates (#2606 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2569 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 14:05:53 +03:00
Yurii Kravets	40c614dc4e	Update vmalert.md (#2580 ) docs: update vmalert/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-05-20 14:02:34 +03:00
Roman Khavronenko	a616e0b997	docs: quickstart update (#2572 ) docs: update docs for beginners QuickStart page was updated with more relevant information. Key Concepts was added to cover basics for the VictoriaMetrics.	2022-05-20 14:00:43 +03:00
Yurii Bychenok	05f80798a5	Updated vmctl documentation, migration from OpenTSDB section (#2595 ) Co-authored-by: Yurii Bychenok <ipeacocks@pm.me>	2022-05-20 13:59:09 +03:00
dependabot[bot]	f3f04c0c8b	build(deps): bump github.com/influxdata/influxdb from 1.9.6 to 1.9.7 (#2589 ) Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.9.6 to 1.9.7. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Changelog](https://github.com/influxdata/influxdb/blob/master/CHANGELOG_OLD.md) - [Commits](https://github.com/influxdata/influxdb/commits) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-05-20 13:55:21 +03:00
Yury Molodov	1e7a6683f8	vmui: optimize data fetching (#2584 )	2022-05-20 13:54:45 +03:00
Yury Molodov	8739fb8a91	vmui: add option to customize url params for individual pages (#2582 )	2022-05-20 13:53:57 +03:00
Roman Khavronenko	87e4e76537	vmalert: support `/rules` path for Grafana's ngalert requests (#2593 ) Unexpectedly, Grafana makes an extra request to `/rules` handler in addition to `/api/v1/rules` calls in alerts UI. This happens only for Grafana versions older than 8.5.*. Apparently, this is related to support of other monitoring systems. Prometheus responds with `text/html` content for UI page `/rules` to such requests. Actually, returning just a blank page with SC=200 works as well. Returning actual response of `/api/v1/rules` results in error in Grafana since it expects a `yaml` (?) in response. So we add a placeholder to `vmalert`. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2583 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 13:53:55 +03:00
Yury Molodov	c8eca88f12	fix: change get display type (#2553 )	2022-05-20 13:47:44 +03:00
Yury Molodov	d13437afd0	vmui: setup predefined dashboards without build (#2541 ) vmui: support predefined dashboards in json format See https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmselect/vmui/dashboards	2022-05-20 13:46:19 +03:00
Roman Khavronenko	ee1f8bae2e	docs: fix liquid syntax errors (#2592 ) For liquid text processor double braces `{{` `}}` are special chars for templating. Since we use them in some of our docs with different purpose, we must escape them to avoid syntax errors from liquid. For escaping curly braces we use bult-in plugin which helps to enclose sections of text via `{% raw %}` and `{% endraw %}`. This approach prevents liquid syntax errors and makes render correct. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 13:42:47 +03:00
Dima Lazerka	6928e5a6d7	Force up nth-check version	2022-05-20 13:41:27 +03:00
Roman Khavronenko	6e5ba1921d	vmalert: fix readme formatting (#2587 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 13:40:41 +03:00
Roman Khavronenko	0790ba5e26	vmalert: follow-up after `0ac1cdfff5` (#2586 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-20 12:25:13 +03:00
Andrii Chubatiuk	7789c47e41	added reusable templates support (#2532 ) Signed-off-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>	2022-05-20 12:25:11 +03:00
Aliaksandr Valialkin	83ff4c411d	app/vmalert: apply `-remoteRead.disablePathAppend` to `-datasource.url` in the same way as for the `-remoteRead.url` This is a follow-up for `0e2486df56` The related pull requests: - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1536 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1712	2022-05-13 16:59:16 +03:00
Denys Holius	b0fc218dc9	Update golangci version to latest v1.46.1 (#2579 )	2022-05-13 16:59:16 +03:00
Aliaksandr Valialkin	c1dd458f56	docs/CHANGELOG.md: document `3f0ecee128` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2577	2022-05-13 16:59:16 +03:00
Roman Khavronenko	2ea625d5bf	vmalert: properly cleanup stale series tracker on rules update (#2577 ) Rules executor within group tracks series sent to remote write in order to mark them as stale if they had disappeared in next evaluation round. The executor uses rules ID as a key to identifies series which belong to rule. On config reload, executor remains active but the set of rules could change. Hence, we need to properly cleanup the tracker for rules which has been disappeared on config reload. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-13 16:59:16 +03:00
Roman Khavronenko	1aa5a19717	vmctl: make linter happy (#2578 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-13 16:59:16 +03:00
Aliaksandr Valialkin	de143323d0	app/{vmagent,vminsert}: mention port 8089 instead of 8189 in the description for -influxListenAddr flag InfluxDB uses 8089 port for sending plain Influx line protocol data over TCP and UDP. See https://docs.influxdata.com/influxdb/v1.8/administration/ports/ This is a follow-up for `20cef877a1`	2022-05-13 16:59:16 +03:00
Yurii Kravets	d55791f59f	Update docs (#2566 ) * deployment/docker: pass `-buildvs=false` to `go build` for production builds This should resolve the `error obtaining VCS status: exit status 128` error when the environment contains incorrect version of git or has incorrect access rights to the directory with VictoriaMetrics source code. See the following links for additional info: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508#issuecomment-1117126702 , - https://github.com/google/ko/issues/672 - https://github.com/golang/go/issues/49004 * lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() This should reduce potential spikes in the number of established connections in the following cases: - when the connection establishing procedure becomes temporarily slow - after a temporary spike in the rate of ConnPool.Get() calls See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552 * docs/CHANGELOG.md: document `c8af625bcc` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322#issuecomment-1120276146 * docs/Cluster-VictoriaMetrics.md: typo fix: `by by` -> `by` * docs: add `resource usage limits` docs, which describe fine-grained tuning for various resource usage limits * docs/Cluster-VictoriaMetrics.md: the `/api/v1/label/.../values` query can take CPU and ram at both vmstorage and vmselect * Update root Readme and root vmagent readme Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-13 16:59:16 +03:00
Aliaksandr Valialkin	116c0b8f2e	docs/vmagent.md: typo fix in the description for `-promscrape.cluster.replicationFactor` command-line flag	2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin	a47717f2ab	docs/Cluster-VictoriaMetrics.md: the `/api/v1/label/.../values` query can take CPU and ram at both vmstorage and vmselect	2022-05-11 20:00:36 +03:00
Aliaksandr Valialkin	7830448a36	docs: add `resource usage limits` docs, which describe fine-grained tuning for various resource usage limits	2022-05-11 19:53:19 +03:00
Aliaksandr Valialkin	6048e2e82c	docs/Cluster-VictoriaMetrics.md: typo fix: `by by` -> `by`	2022-05-11 18:05:05 +03:00
Aliaksandr Valialkin	e35e488220	docs/CHANGELOG.md: document `c8af625bcc` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322#issuecomment-1120276146	2022-05-11 14:33:06 +03:00
Dmytro Kozlov	1a8a24bcb3	vmctl: fix build for solaris os (#2555 ) * vmctl: fix build for solaris os * vmctl: updated dependency (using Syscall instead of Syscall6) * vmctl: updated dependency * vmctl: updated dependency	2022-05-11 14:30:45 +03:00
Denys Holius	f3ac421413	docs: fixed typos in CHANGELOG.md (#2565 )	2022-05-11 14:30:45 +03:00
Aliaksandr Valialkin	d8a276fbe4	lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() This should reduce potential spikes in the number of established connections in the following cases: - when the connection establishing procedure becomes temporarily slow - after a temporary spike in the rate of ConnPool.Get() calls See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552	2022-05-11 14:11:06 +03:00
Aliaksandr Valialkin	0a420c4708	deployment/docker: pass `-buildvs=false` to `go build` for production builds This should resolve the `error obtaining VCS status: exit status 128` error when the environment contains incorrect version of git or has incorrect access rights to the directory with VictoriaMetrics source code. See the following links for additional info: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508#issuecomment-1117126702 , - https://github.com/google/ko/issues/672 - https://github.com/golang/go/issues/49004	2022-05-11 13:57:59 +03:00
Aliaksandr Valialkin	68bfa05437	app/vmselect/promql: do not return values from `label_value()` if the original time series has no values at the selected timestamps	2022-05-09 17:58:39 +03:00
Denys Holius	8752ed9985	Update golangci version to latest v1.46.0 (#2560 ) Update golangci version to latest https://github.com/golangci/golangci-lint/tree/v1.46.0	2022-05-09 17:58:37 +03:00
Aliaksandr Valialkin	d5efd3949b	docs/CHANGELOG.md: document `8f4f5f1d68`	2022-05-09 17:33:04 +03:00
Aliaksandr Valialkin	0d0561ca8c	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:34:56 +03:00
Roman Khavronenko	c9b5e6e8ca	Code check (#2558 ) * vmstorage: make gofmt happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-09 15:29:25 +03:00
Roman Khavronenko	ee93d003d3	Vmalert fix bugs in alerting evaluation (#2557 ) * vmalert: calculate time for firing alert based on the given timestamp Previously, current time was used for checking the `firing` threshold. This is not correct, since alerts are evaluated at specific timestamps. Hence, this specific timestamp supposed to be used in the calculation. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: properly calculate evaluation timestamp for rules Timestamp for rules evaluation should be calculated after the artifical delay for groups start. Otherwise, evaluation timestamp can fall back too far in time. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-09 15:28:30 +03:00
Aliaksandr Valialkin	86d344f6c0	docs/Release-Guide.md: typo fix: signle->single	2022-05-09 15:27:57 +03:00
Manuel Polo	2cc5486248	docs: fix typo in quickstart	2022-05-09 15:27:55 +03:00
Artem Navoiev	7197de0c8c	docs: add flags list to vmbackupmanager (#2554 ) docs: add flags list to vmbackupmanager docs Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: tenmozes <artem@victoriametrics.com>	2022-05-09 15:26:24 +03:00
Marc Hörsken	1580f751d6	app/vmctl: add flag to handle Prometheus remote_write to InfluxDB (#2545 ) Make it possible to migrate timeseries while restoring the original timeseries name previously written from Prometheus to InfluxDB v1 via remote_write. Fixes: https://github.com/VictoriaMetrics/vmctl/issues/8	2022-05-09 15:25:41 +03:00
Aliaksandr Valialkin	e736de8e5c	vendor: `make vendor-update`	2022-05-07 01:49:22 +03:00
Aliaksandr Valialkin	e3819d3e8d	docs/CHANGELOG.md: cut v1.77.1	2022-05-07 01:49:21 +03:00
Aliaksandr Valialkin	e613ca9ba8	app/vminsert/netstorage: re-route samples from readonly vmstorage nodes to healthy nodes if `-dropSamplesOnOverload` command-line flag is set	2022-05-07 01:40:02 +03:00
Aliaksandr Valialkin	e961aec551	app/vmstorage: do not allow to set -retentionPeriod smaller than one day VictoriaMetrics doesn't support retention periods smaller than one day, so do not allow to set it to small values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2496	2022-05-07 00:54:42 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit `67b10896d2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin	994c06cb84	docs/CHANGELOG.md: document `e726340914`	2022-05-06 18:11:16 +03:00
Marc Hörsken	bc28589718	app/vmctl: add flag to skip adding the InfluxDB 'db' label (#2544 ) Make it possible to migrate timeseries without changing labels at all, including not adding the now optional 'db' label.	2022-05-06 18:11:15 +03:00
Dmytro Kozlov	8b5a819266	vmbackup: Prevent save backups to the same folder where TSDB data is (#2547 ) * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip * vmbackup: Prevent save backups to the same folder where TSDB data is * Apply suggestions from code review * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-06 18:04:58 +03:00
Dmytro Kozlov	028c28b84a	deployment/docker: added vmalert.proxyURL flag (#2549 )	2022-05-06 17:34:09 +03:00
Aliaksandr Valialkin	7cd416f5e6	docs/Cluster-VictoriaMetrics.md: typo fix: bandidth -> bandwidth	2022-05-06 16:34:44 +03:00
Roman Khavronenko	fe697a9253	vmctl: add tip about safety flags during for native data export (#2540 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-06 16:28:24 +03:00
Aliaksandr Valialkin	925fa9a7de	docs/Cluster-VictoriaMetrics.md: make the description for `-rpc.disableCompression` command-line flag more clear	2022-05-06 16:24:56 +03:00
Aliaksandr Valialkin	983b465757	app/vmagent: add missing `_total` suffix to `vmagent_remotewrite_global_rows_pushed_before_relabel_total` counter This is a follow up for `c536139d0b`	2022-05-06 15:51:34 +03:00
Aliaksandr Valialkin	af0da45d3e	lib/promscrape: rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics	2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin	2baa9e8f48	app/vmagent: expose `vmagent_remotewrite_global_rows_pushed_before_relabel` and `vmagent_remotewrite_rows_pushed_after_relabel_total` metrics	2022-05-06 15:30:10 +03:00
Aliaksandr Valialkin	9c89ea1c90	app/vmagent: rename `vmagent_remote_write_rate_limit_reached_total` to `vmagent_remotewrite_rate_limit_reached_total` for the sake of consistency with other `vmagent_remotewrite_` metrics	2022-05-06 15:02:18 +03:00
Aliaksandr Valialkin	9d40bb7137	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in `ec2_sd_config` via `az_filters` section	2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin	8babb4aebc	app/vmselect/vmui: `make vmui-update` after `450d879eaa`	2022-05-05 22:11:41 +03:00
Yury Molodov	7b37662a80	vmui: prevent reset relative time (#2543 ) * fix: prevent time picker reset to previous time * fix: add default display type	2022-05-05 21:21:48 +03:00
Yury Molodov	eba02163fe	fix: remove react @types (#2539 )	2022-05-05 21:19:02 +03:00
Denis Fondras	53b68eb254	Fix typo (#2538 )	2022-05-05 21:16:39 +03:00
Roman Khavronenko	c41ae2db2c	vmstorage: switch to rich duration parser for flag `snapshotsMaxAge` (#2542 ) The switch suppose to allow setting `d`, `w`, `y` duration units. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-05 21:13:55 +03:00
Aliaksandr Valialkin	c5e47929c3	docs/Cluster-VictoriaMetrics.md: update cluster scalability tips	2022-05-05 21:09:17 +03:00
Roman Khavronenko	8c30640828	dashboards: bump version requirement for cluster dashboard (#2537 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-05 13:40:07 +03:00
Aliaksandr Valialkin	0634a894a9	docs/CHANGELOG.md: document `bf5e3774cc`	2022-05-05 13:38:44 +03:00
Marc Hörsken	68ca7e3e56	app/vmctl: fix empty/skipped labels after db label (#2536 ) Do not assume the db label to be the last one and also make sure we are not skipping it and everything afterwards. Breaking the loop would cause following labels to be empty.	2022-05-05 13:38:43 +03:00
Aliaksandr Valialkin	2ce1d09135	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call Previously filters wheren't passed to this call after the commit `0e09fdb8b0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin	873f55bac5	lib/awsapi: pass `filtersQueryString` arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: `0e09fdb8b0` `d289ecded1`	2022-05-05 10:29:47 +03:00
Aliaksandr Valialkin	0ad5b64930	docs/CHANGELOG.md: cut v1.77.0	2022-05-05 00:16:31 +03:00
Aliaksandr Valialkin	adf29d048c	vendor: `make vendor-update`	2022-05-04 23:56:45 +03:00
Aliaksandr Valialkin	3fee369997	app/vmselect: follow-up after `8639e79d38`	2022-05-04 23:41:00 +03:00
Aliaksandr Valialkin	2213eef6b7	docs/vmbackup.md: added missing -storageDataPath argument in the command for creating daily backups	2022-05-04 22:50:18 +03:00
Aliaksandr Valialkin	4ae1a8514a	docs/vmbackup.md: acutalize docs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2503	2022-05-04 22:44:40 +03:00
Dmytro Kozlov	4f40dc9829	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535 ) * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:48 +03:00
Aliaksandr Valialkin	9fea63d1c8	app/vmagent: rename -remoteWrite.useSigv4 command-line flag to -remoteWrite.aws.useSigv4, so its name is consistent with the other -remoteWrite.aws.* command-line flags	2022-05-04 20:41:45 +03:00
Aliaksandr Valialkin	358fa99af2	app/vmalert: run `make quicktemplate-gen` from the root directory after the commit `f6dcfbcdd6`	2022-05-04 20:28:37 +03:00
Nikolay	7e58cba6cf	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458 ) * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:28:37 +03:00
Dmytro Kozlov	0aeefeb5f1	vmalert/tpl: fixed truncating alerts expression in table (#2494 ) vmalert: improve `/groups` UI visual The change also fixes truncated rules expressions in UI https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2484	2022-05-04 20:28:37 +03:00
Aliaksandr Valialkin	a297315ff1	docs/guides/multi-regional-setup-dedicated-regions.md: clarify wording on vmagent configuration	2022-05-04 18:50:37 +03:00
Aliaksandr Valialkin	4dfd84c505	docs/Cluster-VictoriaMetrics.md: move `environment variables` entry closer to `cluster setup` section	2022-05-04 18:41:44 +03:00
Aliaksandr Valialkin	9dc9e03a14	docs/CHANGELOG.md: yet another typo fix: present -> pressed	2022-05-04 18:20:58 +03:00
Aliaksandr Valialkin	c2610e4186	docs/CHANGELOG.md: typo fixes	2022-05-04 18:19:05 +03:00
Aliaksandr Valialkin	47010a9875	docs/CHANGELOG.md: document `8639e79d38`	2022-05-04 10:46:32 +03:00
Nikolay	48041305a2	app/vmselect: adds proxy for rules API (#2516 ) * app/vmselect: adds proxy for rules and alerts API It allows to visualization for rules at grafana https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739 * Update app/vmselect/main.go Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-05-04 10:12:15 +03:00
Dima Lazerka	b5fce4190c	Code cleanup (#343 ) * Small code cleanup: remove Request from params * Extract common params to all export handlers * Renamed ExportParams -> exportParams * wip Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-03 15:53:58 +03:00
Aliaksandr Valialkin	4fa3cd701c	docs/Cluster-VictoriaMetrics.md: typo fix: serparated -> separated	2022-05-03 15:20:11 +03:00
Aliaksandr Valialkin	25266d2194	docs/CHANGELOG.md: document `3575aabeaf`	2022-05-03 14:01:58 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530 ) standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Aliaksandr Valialkin	ec3a37896f	all: add -cluster.tlsInsecureSkipVerify command-line option to vminsert, vmselect and vmstorage components in order to be able to disable TLS certificate verification in mTLS mode See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2490	2022-05-03 13:13:43 +03:00
Aliaksandr Valialkin	7291c81f0d	docs/Cluster-VictoriaMetrics.md: refer to the doc on how to set up mTLS	2022-05-03 11:54:05 +03:00
Aliaksandr Valialkin	ccf44e810c	docs/CHANGELOG.md: document `488c34f5e1`	2022-05-03 11:01:27 +03:00
Aliaksandr Valialkin	75bd7f1539	app/vmui: fix up/down arrow keys on multi-line query after `a186434b50`	2022-05-03 10:51:08 +03:00
Dmytro Kozlov	b9ff61811d	vmctl: fixed blocking when aborting import process (#2509 ) vmctl: fix vmctl blocking on process interrupt This change prevents vmctl from indefinite blocking on receiving the interrupt signal. The update touches all import modes and suppose to improve tool reliability. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2491	2022-05-03 10:51:06 +03:00
Aliaksandr Valialkin	d384997657	docs/CHANGELOG.md: document `d0706c8c95`	2022-05-02 22:25:47 +03:00
Gard Rimestad	ceadcb7f8e	app/vmagent add metric for rate limit (#2521 ) This adds a metric for the rate limit. The limit is present as a flag currently: `flag{name="remoteWrite.rateLimit", value="500000", is_set="true"} 1` We are running many instances of vmagent and when creating alerts it is harder than it needs to be when extracting the value from the flag. With this change it should be easier to monitor how close to the limit we are. `((100/vmagent_remotewrite_rate_limit{account="account"})*sum (rate(vmagent_remotewrite_conn_bytes_written_total{account="account"}))) and ON (account) flag{name="remoteWrite.rateLimit"} == 1`	2022-05-02 22:25:47 +03:00
Aliaksandr Valialkin	361b08c30e	lib/storage: leave the last sample per each discrete interval during the deduplicaton This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin	7ca32c21c8	app/vmui: execute query by pressing `enter` in the same way as Prometheus does Multi-line query can be entered via `shift-enter` in the query input field	2022-05-02 20:24:43 +03:00
Yury Molodov	6292185abf	vmui: support node v.18 (#2529 ) * fix: add support vmui node18 * fix: remove @mui/styles (legacy styling solution) * Update app/vmui/Dockerfile-build * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 20:24:42 +03:00
Aliaksandr Valialkin	b06b02c7e3	vendor: `make vendor-update`	2022-05-02 16:01:06 +03:00
Aliaksandr Valialkin	58e4e16588	docs/Cluster-VictoriaMetrics.md: move here the `deduplication` docs related to cluster version	2022-05-02 15:54:48 +03:00
Max Golionko	be81a0a327	added details about deduplication (#2527 ) * added details about deduplication * Update docs/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Apply suggestions from code review * Update docs/README.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-05-02 15:54:46 +03:00
Aliaksandr Valialkin	a70ac35ac7	docs/CHANGELOG.md: document `3616337812` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2514	2022-05-02 15:37:54 +03:00
Aliaksandr Valialkin	75f4adab40	docs/CHANGELOG.md: document `32a6b67e6c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1761	2022-05-02 15:37:54 +03:00
Aliaksandr Valialkin	693e1838b3	docs/CHANGELOG.md: document `b2294d1cf1`	2022-05-02 15:37:54 +03:00
Aliaksandr Valialkin	725cb64e81	app/vmalert: run `make quicktemplate-gen` from the repository root This is a follow-up after `b2294d1cf1`	2022-05-02 15:37:54 +03:00
Dmytro Kozlov	4764f6e522	vmalert: added disableProgressBar flag which disable progressbar (#2506 ) vmalert: added disableProgressBar flag which disable progressbar https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1761	2022-05-02 15:37:54 +03:00
Roman Khavronenko	d0d0be9031	vmalert: do not execute templates during validation (#2528 ) Function `ValidateTemplates`, used on the vmalert startup, is supposed to check whether used templates and functions in loaded rules are correct. The function was parsing and executing loaded templates. However, rules may contain functions which can't be executed without values (label values or query results), like `slice`. Because of this, validation for completely valid expression `{{ slice $labels.job 9 }}` will fail since `$labels.job` is empty during validation. This PR updates `ValidateTemplates` function to only parse templates without executing them. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2514 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-05-02 15:37:54 +03:00
Aliaksandr Valialkin	190c8b463c	lib/netutil: close connections in ConnPool if they are idle for more than 30 seconds Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-05-02 15:01:52 +03:00
Artem Navoiev	11db05a4ff	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487 ) * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:24:12 +03:00
Aliaksandr Valialkin	a436836402	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:58:08 +03:00
Dmytro Kozlov	25e54d2b50	vmctl/vm: added datapoints collection bar (#2486 ) add progress bars to the VM importer The new progress bars supposed to display the processing speed per each VM importer worker. This info should help to identify if there is a bottleneck on the VM side during the import process, without waiting for its finish. The new progress bars can be disabled by passing `vm-disable-progress-bar` flag. Plotting multiple progress bars requires using experimental progress bar pool from github.com/cheggaaa/pb/v3. Switch to progress bar pool required changes in all import modes. The openTSDB mode wasn't changed due to its implementation, which implies individual progress bars per each series. Because of this, using the pool wasn't possible. Signed-off-by: dmitryk-dk <kozlovdmitriyy@gmail.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2022-05-02 10:58:06 +03:00
Peter Dupej	9e193fb764	Update MetricsQL.md (#2519 ) remove typo from label_replace doc.	2022-05-02 09:43:57 +03:00
Aliaksandr Valialkin	f1d31b66b9	vendor: update github.com/valyala/gozstd from v1.16.0 to v1.17.0	2022-04-29 19:35:07 +03:00
Aliaksandr Valialkin	eaae544238	docs/CHANGELOG.md: document `c7aad8d441`	2022-04-29 13:02:43 +03:00
Nikolay	022b2875ad	app/vmselect: adds API /api/v1/status/buildinfo (#2515 ) * app/vmselect: adds API /api/v1/status/buildinfo it should fix an compability error with grafana 8.5 prometheus datasource https://github.com/grafana/grafana/pull/46771 * Update main.go	2022-04-29 13:02:43 +03:00
Aliaksandr Valialkin	90f9ed260e	app/vmselect/vmui: `make vmui-update` after `da04e9d1de`	2022-04-29 13:02:43 +03:00
Vitaliy Vasilenko	d7e1971042	vmui: fix default server path (#2511 )	2022-04-29 13:02:43 +03:00
Dima Lazerka	e6ee235707	Export "null" in jsonl instead of NaN (#2518 ) * Export "null" in jsonl instead of NaN The NaN appeared because of staleness markers that were added for compatibility. I think it's better to use json `null`, implemented here. Also maybe it also makes sense to add a flag like `?skip-staleness-markers=true` to `/export`, to skip nulls at all? * Update app/vmselect/prometheus/export.qtpl * app/vmselect/prometheus/export.qtpl.go: `make quicktemplate-gen` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-29 12:51:03 +03:00
Aliaksandr Valialkin	5f41c48e4f	deployment/docker/docker-compose.yml: update Grafana version from v8.3.5 to v8.5.1	2022-04-29 12:02:32 +03:00
Aliaksandr Valialkin	2130f3cf05	app/vmui/Dockerfile-build: fix dependency to nodejs v17, since vmui doesnt work with nodejs v18	2022-04-29 11:18:16 +03:00
Dima Lazerka	837e440865	Fix targetstatus qtpl paths (#2517 ) Ran `make quicktemplate-gen` from the root directory	2022-04-29 11:18:14 +03:00
Aliaksandr Valialkin	83db416567	docs/MetricsQL.md: clarify keep_metric_names docs	2022-04-27 11:25:02 +03:00
Yury Molodov	dac203320a	vmui: expression alias (#2495 ) * feat: add alias for queries * docs: update docs for predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-27 10:59:51 +03:00
Aliaksandr Valialkin	6eb1580158	app/vmselect/promql: add `tlast_change_over_time(m[d])` function, which returns the timestamp for the last change of `m` on the given lookbehind window `d`	2022-04-27 10:58:40 +03:00
Aliaksandr Valialkin	917bb702ef	app/vmagent: substitute hard-to-read 500000000 with 500MB in `-remoteWrite.maxDiskUsagePerURL` description	2022-04-26 15:48:44 +03:00
Yury Molodov	eae6f68be2	vmui: add support relative time (#2504 ) * feat: add support relative time * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-26 15:46:47 +03:00
Aliaksandr Valialkin	aa82987d70	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin	155cd5d6e1	vendor: `make vendor-update`	2022-04-26 15:27:40 +03:00
Aliaksandr Valialkin	7d9d790bfb	docs/Cluster-VictoriaMetrics.md: remove incorrect and misleading instructions for passing `-replicationFactor` flag to vmselect nodes in multi-level setup. The `-replicationFactor` passed to top-level `vmselect` nodes mustn't exceed the `-replicationFactor` passed to top-level `vminsert` nodes	2022-04-26 15:13:08 +03:00
Aliaksandr Valialkin	cd4d1599cb	docs/CHANGELOG.md: document `4c1fbcd6b0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2368	2022-04-26 15:09:57 +03:00
hagen1778	e856d74b7b	dashboards: replace fixed interval of `5m` for `rate` expressions Before we used fixed `5m` interval for expressions with `rate` func. Unfortunately, this interval wasn't a fit for all the cases. So we switch to `$__rate_interval` instead. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
hagen1778	16b3374874	dashboards: add new panel `IndexDB items rate` The new panel supposed to reflect the pressure on indexDB caused by churn rate or new series registration. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
hagen1778	1762256c7e	dashboards: mention that `Rows.Sent` can be affected by replication Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
hagen1778	4255cb7559	dashboards: rm "Deferred merges" panel since it could be misleading See more context here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1682#issuecomment-938608067 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
hagen1778	0bbc7221f3	dashboards: add adhoc filter to dasbhoard variables The adhoc filter allows to quickly apply global filters without modifying the panels. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
hagen1778	80e8413f3a	dashboards: remove index filter from stats panel for DiskUsage The diskUsage stats panel was showing disk usage without including size of the index, which is not correct. The filter was removed to reflect the total disk usage. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2368 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-24 23:25:32 +03:00
Aliaksandr Valialkin	374212333f	docs/CHANGELOG.md: typo fix: `may result` -> `could result`	2022-04-23 00:31:47 +03:00
Aliaksandr Valialkin	a85ef60b4b	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:24:13 +03:00
Aliaksandr Valialkin	4c3cd96db5	lib/promauth: add support for `min_version` option at `tls_config` section in the same way as Prometheus does	2022-04-23 00:24:11 +03:00
Aliaksandr Valialkin	808a2f3b61	lib/promauth: add support for `proxy_url` option at `oauth2` section in the same way as Prometheus does	2022-04-23 00:01:53 +03:00
Aliaksandr Valialkin	4ade8511e2	lib/promauth: add support for `tls_config` section at `oauth2` config in the same way as Prometheus does	2022-04-23 00:01:52 +03:00
Aliaksandr Valialkin	c2b13e6a04	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin	a89e31b304	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin	cc6eae6992	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin	60f74dab56	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated This is a follow-up after `26b78ad707`	2022-04-22 13:23:37 +03:00
Aliaksandr Valialkin	2c7f7799eb	vendor: `make vendor-update`	2022-04-21 16:01:19 +03:00
Aliaksandr Valialkin	a69dabf709	docs/Cluster-VictoriaMetrics.md: mention that enterprise binaries are available for evaluation	2022-04-21 15:57:13 +03:00
Aliaksandr Valialkin	c341a46745	docs/Single-server-VictoriaMetrics.md: refer to the docs on how to set up multiple vmagent instances for scraping the same set of targets	2022-04-21 15:52:22 +03:00
Aliaksandr Valialkin	dac24aa342	app/vmselect/promql: properly handle `scalar default vector`, `scalar if vector` and `scalar ifnot vector` queries Previously `vector` time series could be unexpectedly returned from such queries	2022-04-21 15:34:14 +03:00
Aliaksandr Valialkin	bc54ae9608	app/vmselect/promql: fix comparison to nan The comparison to nan has been broken in `d335cc886c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2022-04-21 14:56:41 +03:00
Aliaksandr Valialkin	a5cfe5d13e	app/vmselect/promql: add `drop_common_labels()` function	2022-04-21 14:20:36 +03:00
Aliaksandr Valialkin	ed1b394a1a	app/vmstorage: expose `vm_indexdb_items_added_total` and `vm_indexdb_items_added_size_bytes_total` counters at `/metrics` page These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin	fea9d1e6ee	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for `2341bd48d7`	2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin	7ca08945fd	app/vmselect/promql: fix `q default b` where `b` may have empty time series	2022-04-21 11:43:09 +03:00
Aliaksandr Valialkin	25b841c6ed	app/vmselect/promql: fix `duplicate time series` error on joins against time series filtered by values This should prevent from `duplicate time series` errors when executing the following query: kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending\|Running"}==1) where `kube_pod_status_phase{phase=~"Pending\|Running"}==1` filters out diplicate time series	2022-04-20 22:21:20 +03:00
Aliaksandr Valialkin	0ef7a05fc0	app/vmselect/promql: rename removeNaNs() to more clear removeEmptySeries()	2022-04-20 19:53:24 +03:00
Aliaksandr Valialkin	1e0517b9cd	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:41:23 +03:00
Aliaksandr Valialkin	1ae16bf671	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:27 +03:00
Aliaksandr Valialkin	e9f08b1e6a	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin	90db923662	docs/CHANGELOG.md: document that the service discovery speed now scales with the number of CPU cores	2022-04-20 16:22:50 +03:00
Aliaksandr Valialkin	909a3ee0e4	lib/promscrape: follow-up after `91e290a8ff`	2022-04-20 16:12:26 +03:00
Nikolay	429848a67d	lib/promscrape: reduce latency for k8s GetLabels (#2454 ) replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:12:25 +03:00
Dmytro Kozlov	9dbfd99777	lib/promscrape: simply update UI (#2479 ) * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 15:38:04 +03:00
Aliaksandr Valialkin	45385a5dc6	lib/promscrape: optimize getScrapeWork() function Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:34:18 +03:00
Aliaksandr Valialkin	bfa0b8f710	lib/promscrape: use a hash over target labels as a key for dropped targets' map This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:23:54 +03:00
Aliaksandr Valialkin	d0bac8e224	all: typo fix: Kuberntes -> Kubernetes	2022-04-20 10:51:41 +03:00
Dmytro Kozlov	17552dba8b	lib/promscrape: Enable filters for endpoint and labels (#2466 ) * lib/promscrape: Enable filters for endpoint and labels * lib/promscrape: cleanup * lib/promscrape: update template * lib/promscrape: move logic filter logic to backend * lib/promscrape: updated placeholder * lib/promscrape: updated placeholder * lib/promscrape: use two different fields for filters, updated form, added error on parsing queries * lib/promscrape: rename functions * lib/promscrape: removed unused values * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-19 18:27:44 +03:00
Aliaksandr Valialkin	7a622a71ea	vendor: `make vendor-update`	2022-04-19 11:43:02 +03:00
naveensrinivasan	21f80fa137	chore: Set permissions for GitHub actions Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much. - Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs [Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>	2022-04-19 11:33:48 +03:00
Nikolay	628905f080	lib/promscrape: adds job restart method (#2455 ) * lib/promscrape: adds job restart method it must restart only ScrapeConfig with changed content this change greatly reduce time, that needed for job restart and it should decrease possible data loss when config frequently changed at kubernetes based deployments Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 20:29:33 +03:00
Dima Lazerka	6e4d84c1f1	Add GitHub workflow for code scanning (#2453 ) Add pre-generated workflow definition for GitHub's CodeQL code scanning.	2022-04-16 19:01:37 +03:00
Yury Molodov	60c4e0022a	fix: prevent graph hiding without data (#2456 ) * fix: prevent graph hiding without data * fix: add yaxis labels default * app/vmselect: `make vmui-update` * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 17:16:17 +03:00
Aliaksandr Valialkin	944070eafa	docs/Cluster-VictoriaMetrics.md: sync docs	2022-04-16 16:59:42 +03:00
Aliaksandr Valialkin	7debf57ca6	lib/httpserver: clarify that `-tls` flag enables TLS for http requests to `-httpListenAddr`	2022-04-16 16:59:41 +03:00
Aliaksandr Valialkin	a7689e1b0c	app/vmstorage: add support for mTLS cipher suites via `-cluster.tlsCipherSuites` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404	2022-04-16 16:36:38 +03:00
Aliaksandr Valialkin	0156a8b1e1	docs/Cluster-VictoriaMetrics.md: update docs after `26ae50ec26`	2022-04-16 16:04:32 +03:00
Aliaksandr Valialkin	6bd032a6d3	docs: sync docs with the latest changes	2022-04-16 16:00:27 +03:00
Aliaksandr Valialkin	27e74f25d6	lib/httpserver: follow up after `def0032c7d`	2022-04-16 15:52:44 +03:00
Dmytro Kozlov	26ae50ec26	lib/httpserver: added tlsCipherSuites flag (#2468 ) * lib/httpserver: added tlsCipherSuites flag * lib/httpserver: compare lower case strings * lib/httpserver: use EqualFold * lib/httpserver: used flagutil.NewArray, supported only strings cipher suites * lib/httpserver: updated flag description, added flag to documentation * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 15:52:42 +03:00
Aliaksandr Valialkin	c50e48a74c	lib/promscrape: follow-up after `baa1c24b36`	2022-04-16 14:26:38 +03:00
Nikolay	a56ee034af	lib/promscrape: removes omitempty for ScrapeConfig (#2457 ) This change fixes incorrect marshalling for ScrapeConfig it affects http endpoint and ScrapeConfig checksum. With omitempty, custom Marshaller is not called if field is not a pointer. Previously this issue happened at vmalert	2022-04-16 14:26:36 +03:00
Aliaksandr Valialkin	f6fed6637d	docs/Cluster-VictoriaMetrics.md: clarify the issue with multi-level cluster, which may result in data gaps at some AZs	2022-04-16 13:03:39 +03:00
Aliaksandr Valialkin	ef86cbabcc	docs/Cluster-VictoriaMetrics.md: improve docs on cluster setup, resizing and scalability	2022-04-15 15:04:10 +03:00
Aliaksandr Valialkin	437a12cb64	docs/Cluster-VictoriaMetrics.md: mention about possible issues with multi-level cluster setup	2022-04-15 14:26:01 +03:00
Aliaksandr Valialkin	564996da14	docs/CHANGELOG.md: document `45fcaa33e8`	2022-04-13 14:14:25 +03:00
Aliaksandr Valialkin	951b2a0067	docs/CHANGELOG.md: document `f7e4c5a628`	2022-04-13 14:14:25 +03:00
Aliaksandr Valialkin	4fc679c9cb	deployment/docker: update Go builder from go1.18.0 to go1.18.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.18.1+label%3ACherryPickApproved	2022-04-13 14:14:25 +03:00
Dmytro Kozlov	1b9d82a9e4	vmctl: Return non zero error code if validation or subcommand fails (#2462 )	2022-04-13 14:14:25 +03:00
Roman Khavronenko	56cd2b918a	vmalert: add DNS service discovery (#2465 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2460 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-13 14:14:25 +03:00
Anton Bystrov	af96e6594c	Update CHANGELOG.md (#2463 ) May be mispint here?	2022-04-13 14:14:25 +03:00
Aliaksandr Valialkin	4a6b56e5be	docs/CHANGELOG.md: cut v1.76.1	2022-04-12 16:21:06 +03:00
Aliaksandr Valialkin	3e2a4c07cd	app/vmui: further improvements for number display on graphs This is a follow-up for `c4d2cd8336` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2409	2022-04-12 16:02:37 +03:00
Aliaksandr Valialkin	50a6354a03	docs/CHANGELOG.md: link to the bug related to improper handling of maxSeries limit passed from vmselect to vmstorage	2022-04-12 16:02:37 +03:00
Dmytro Kozlov	19538b727e	vmui: changed function (#2451 ) * vmui: fixed yaxis labels * vmui: changed function	2022-04-12 16:02:37 +03:00
Dmytro Kozlov	46a1e28ed7	vmui: fixed yaxis labels (#2448 )	2022-04-12 16:02:37 +03:00
Aliaksandr Valialkin	4197c61ef8	app/vmselect: `make vmui-update`	2022-04-12 14:35:37 +03:00
Roman Khavronenko	1e9ba4d133	github/dependabot.yml: disable versions update for vmui (#2449 ) The change disables versions autopupdate for vmui package. The change has no impact on security updates, which have a separate, internal limit of ten open pull requests. See https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#open-pull-requests-limit Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-12 14:27:38 +03:00
Yurii Kravets	488d300d8b	Update Quick-Start (#2422 ) * Update Quick-Start * Update docs/Quick-Start.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update docs/Quick-Start.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update docs/Quick-Start.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Update docs/Quick-Start.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * update Quick-Start.md added "Starting VM-Cluster via Docker" + Anchor fixes * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-12 14:27:36 +03:00
Aliaksandr Valialkin	1bc1f9bc2a	app/vmui: revert back incompatible changes proposed by dependabot at `da6a1642e0` and further commits	2022-04-12 14:08:42 +03:00
dependabot[bot]	97e59de7c3	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2447 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.17.0 to 5.19.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.19.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:27:02 +03:00
dependabot[bot]	c7ba233869	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#2446 ) Bumps [@mui/styles](https://github.com/mui/material-ui/tree/HEAD/packages/mui-styles) from 5.5.3 to 5.6.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.6.1/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:24:15 +03:00
dependabot[bot]	c4645f7823	build(deps): bump @testing-library/jest-dom in /app/vmui/packages/vmui (#2445 ) Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.16.3 to 5.16.4. - [Release notes](https://github.com/testing-library/jest-dom/releases) - [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/jest-dom/compare/v5.16.3...v5.16.4) --- updated-dependencies: - dependency-name: "@testing-library/jest-dom" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:22:34 +03:00
dependabot[bot]	8280fb5092	build(deps): bump @testing-library/user-event in /app/vmui/packages/vmui (#2442 ) Bumps [@testing-library/user-event](https://github.com/testing-library/user-event) from 14.0.4 to 14.1.0. - [Release notes](https://github.com/testing-library/user-event/releases) - [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/user-event/compare/v14.0.4...v14.1) --- updated-dependencies: - dependency-name: "@testing-library/user-event" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:21:12 +03:00
dependabot[bot]	8d16aacc9a	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2441 ) Bumps [@mui/icons-material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-icons-material) from 5.5.1 to 5.6.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.6.1/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:21:12 +03:00
dependabot[bot]	c4a318b91b	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2444 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.5.3 to 5.6.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.6.1/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:21:12 +03:00
dependabot[bot]	ace4d51ede	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2440 ) Bumps [@mui/lab](https://github.com/mui/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.75 to 5.0.0-alpha.77. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:17:31 +03:00
dependabot[bot]	5bf48f8285	build(deps): bump @testing-library/react in /app/vmui/packages/vmui (#2438 ) Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 13.0.0 to 13.0.1. - [Release notes](https://github.com/testing-library/react-testing-library/releases) - [Changelog](https://github.com/testing-library/react-testing-library/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/react-testing-library/compare/v13.0.0...v13.0.1) --- updated-dependencies: - dependency-name: "@testing-library/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:17:31 +03:00
dependabot[bot]	d3695ad045	build(deps-dev): bump @typescript-eslint/parser (#2443 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.17.0 to 5.19.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.19.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:17:31 +03:00
dependabot[bot]	af645849ec	build(deps): bump @types/react in /app/vmui/packages/vmui (#2439 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.43 to 18.0.2. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:13:33 +03:00
dependabot[bot]	d12798b266	build(deps): bump preact in /app/vmui/packages/vmui (#2431 ) Bumps [preact](https://github.com/preactjs/preact) from 10.7.0 to 10.7.1. - [Release notes](https://github.com/preactjs/preact/releases) - [Commits](https://github.com/preactjs/preact/compare/10.7.0...10.7.1) --- updated-dependencies: - dependency-name: preact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:13:33 +03:00
dependabot[bot]	1b77564bb6	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2433 ) Bumps [@mui/icons-material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-icons-material) from 5.5.1 to 5.6.0. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.6.0/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:09:21 +03:00
dependabot[bot]	71613e1d0b	build(deps): bump @types/react-dom in /app/vmui/packages/vmui (#2430 ) Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 17.0.14 to 18.0.0. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom) --- updated-dependencies: - dependency-name: "@types/react-dom" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:09:21 +03:00
dependabot[bot]	b0d04d1843	build(deps): bump marked in /app/vmui/packages/vmui (#2429 ) Bumps [marked](https://github.com/markedjs/marked) from 4.0.12 to 4.0.14. - [Release notes](https://github.com/markedjs/marked/releases) - [Changelog](https://github.com/markedjs/marked/blob/master/.releaserc.json) - [Commits](https://github.com/markedjs/marked/compare/v4.0.12...v4.0.14) --- updated-dependencies: - dependency-name: marked dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-12 13:09:21 +03:00
Roman Khavronenko	e48a59eaa6	docs: add managed vm documentation section (#2437 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-12 13:04:11 +03:00
Aliaksandr Valialkin	302464d27d	deployment/docker: update base Docker image from Alpine 3.15.3 to Alpine 3.15.4 See https://alpinelinux.org/posts/Alpine-3.12.12-3.13.10-3.14.6-3.15.4-released.html	2022-04-12 13:01:56 +03:00
Aliaksandr Valialkin	4a3172f150	lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* This allows Go complier to generate more optimal code without bound checks	2022-04-12 12:56:52 +03:00
Aliaksandr Valialkin	b9e133678f	vendor: `make vendor-update`	2022-04-12 12:56:50 +03:00
Aliaksandr Valialkin	70ad171070	lib/promscrape: follow-up after `7e79adfb55`	2022-04-12 12:37:03 +03:00
Nikolay	e26bcb8bbb	lib/promscrape: allows to use k8s pod name as clusterMemberNum (#2436 ) * lib/promscrape: allows to use k8s pod name as clusterMemberNum it must improve user expirience and simplify clustering scrapers. it must allow to use vmagent cluster with distroless images https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-12 12:37:02 +03:00
Yurii Kravets	e5b8a5210e	url-examples.md (#2435 ) * url-examples.md added 1 more example * Update url-examples.md	2022-04-12 11:30:53 +03:00
Aliaksandr Valialkin	81b7a31cb1	app/vmstorage: properly handle `maxSeries` limit passed from vmselect to vmstorage	2022-04-12 11:19:07 +03:00
Aliaksandr Valialkin	e3bf464f11	lib/protoparser/native: follow-up after `fe01f4803d`	2022-04-11 19:27:53 +03:00
Nikolay	39225fc809	lib/protoparser/native: fixes parseStream dead-lock (#2423 ) previously, if native block cannot be unmarshaled, wg.Done wasn't called by unmarshal work. It leads to connection blocking and possible dead-lock at client side	2022-04-11 19:27:51 +03:00
Aliaksandr Valialkin	b3a3e9990f	vendor: update github.com/VictoriaMetrics/metricsql from v0.40.0 to v0.41.0 This allows using built-in function names as with template names	2022-04-11 18:32:18 +03:00
Aliaksandr Valialkin	5b265b3031	docs/Single-server-VictoriaMetrics.md: clarify that `ingestion protocol` means `data ingestion protocol`	2022-04-11 12:57:55 +03:00
Aliaksandr Valialkin	3605e1743b	app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests	2022-04-11 12:32:23 +03:00
Aliaksandr Valialkin	3c27bde77e	docs/CHANGELOG.md: document `ed364a42e3`	2022-04-11 12:12:07 +03:00
hagen1778	fa9601a0f1	vmalert: support relabeling for alert labels sent via notifier Before, relabeling for notifier configured via file was supported only for target labels discovered via SD. With this change, new config field `alert_relabel_configs` is introduced for applying relabeling to labels of sent alerts. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-11 12:12:04 +03:00
Aliaksandr Valialkin	e1b274d26c	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics supports data ingestion in Graphite protocol at `Graphite API usage` chapter	2022-04-10 16:21:19 +03:00
Aliaksandr Valialkin	cf9d261038	docs/Cluster-VictoriaMetrics.md: update docs after `b843f0e229`	2022-04-10 16:17:57 +03:00
Yurii Kravets	ace7dfe5d1	Update url-examples (#2410 )	2022-04-08 13:07:06 +03:00
Aliaksandr Valialkin	4f5622b6fa	docs/Cluster-VictoriaMetrics.md: clarify high availability docs	2022-04-08 12:51:49 +03:00
Aliaksandr Valialkin	a4a15a462b	app/vmselect/netstorage: bump RPC API versions for vmselect->vmstorage communications This is a follow-up after `b843f0e229`	2022-04-08 12:36:04 +03:00
Aliaksandr Valialkin	ce02d086d0	docs/CHANGELOG.md: document backwards-incompatible changes in cluster version of v1.76.0	2022-04-08 12:06:49 +03:00
Dmytro Kozlov	2aeb221691	docs/guides: Multi-regional setup with VictoriaMetrics (#2416 ) * docs/guides: Multi-regional setup with VictoriaMetrics * docs/guides: cleanup	2022-04-08 11:49:04 +03:00
Aliaksandr Valialkin	d7557b12ab	docs/CHANGELOG.md: document the bugfix in hitCount function	2022-04-08 11:48:36 +03:00
Aliaksandr Valialkin	949bf25a87	docs/CHANGELOG.md: typo fix	2022-04-07 17:20:20 +03:00
Aliaksandr Valialkin	e59019d088	docs/CHANGELOG.md: cut v1.76.0	2022-04-07 15:34:12 +03:00
Aliaksandr Valialkin	f1c01a6fcc	vendor: `make vendor-update`	2022-04-07 15:28:44 +03:00
Aliaksandr Valialkin	edb139cfe4	lib/memory: export `process_memory_limit_bytes` metric, which shows the amounts of memory the current process has access to This metric is equivalent to `vm_available_memory_bytes`, but it has better name, since the metric is related to a process, not VictoriaMetrics itself. Leave `vm_available_memory_bytes` for backwards compatibility.	2022-04-07 15:24:08 +03:00
Roman Khavronenko	4de1b2b74a	vmalert: fix labels and annotations processing for alerts (#2403 ) To improve compatibility with Prometheus alerting the order of templates processing has changed. Before, vmalert did all labels processing beforehand. It meant all extra labels (such as `alertname`, `alertgroup` or rule labels) were available in templating. All collisions were resolved in favour of extra labels. In Prometheus, only labels from the received metric are available in templating, so no collisions are possible. This change makes vmalert's behaviour similar to Prometheus. For example, consider alerting rule which is triggered by time series with `alertname` label. In vmalert, this label would be overriden by alerting rule's name everywhere: for alert labels, for annotations, etc. In Prometheus, it would be overriden for alert's labels only, but in annotations the original label value would be available. See more details here https://github.com/prometheus/compliance/issues/80 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-07 15:24:06 +03:00
Aliaksandr Valialkin	cb319b15bb	lib/storage: increase the number of rawRowsShard shards on systems with more than 4 CPU cores This should improve data ingestion scalability on systems with many CPU cores	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	8ef9348801	lib/mergeset: use more rawItemsShard shards on multi-CPU systems This should improve the scalability for registering of new time series on multi-CPU system	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	db00ddd23e	lib/mergeset: skip common prefixes when comparing inmemoryBlock items This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData if they have common prefix. While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items. This should improve performance for inmemoryBlock.MarshalSortedData during background merge.	2022-04-06 18:55:25 +03:00
Aliaksandr Valialkin	88c2631320	lib/protoparser: remove superflowous memory allocations during protocol parsing	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	a57d5dd415	app/vmagent: properly initialize stdDialer This is a follow-up commit for `7da20a4b3f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	123a88bb65	lib/storage: reuse sync.WaitGroup objects This reduces GC load by up to 10% according to memory profiling	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	f526c7814e	lib/cgroup: reduce the default GOGC value from 50% to 30% This reduces memory usage under production workloads by up to 10%, while CPU spent on GC remains roughly the same. The CPU spent on GC can be monitored with go_memstats_gc_cpu_fraction metric	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	f082e64e0c	app/vmagent: reduce the probability of TLS handshake timeout when dialing the remote storage The following actions are taken: - Increase the TLS hashdshake timeout from 5 seconds to 10 seconds - Increase dial timeout from 5 seconds to 30 seconds - Specify DialContext instead of Dial in http.Transport. This allows properly handling the Context arg during dialing the remote storage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1699	2022-04-06 12:35:14 +03:00
Aliaksandr Valialkin	087609a12b	docs/Release-Guide.md: add missing steps	2022-04-06 11:41:42 +03:00
Aliaksandr Valialkin	0f1ebd911d	lib/workingsetcache: reuse prev cache after its reset This should reduce memory churn rate	2022-04-05 20:39:44 +03:00
Aliaksandr Valialkin	89c550fa5d	vendor: update github.com/VictoriaMetrics/fastcache from v1.9.0 to v1.10.0	2022-04-05 20:39:43 +03:00
Aliaksandr Valialkin	c24c5b8926	docs/CHANGELOG.md: document `0c0efc7781`	2022-04-05 19:22:08 +03:00
Aliaksandr Valialkin	ac93c36be7	lib/workingsetcache: check more frequently for cache size overflow This should reduce the probability of cache size limit overflow	2022-04-05 18:05:33 +03:00
Nikolay	7eb49d204f	vmctl verify-blocks command (#2390 ) * lib/protoparser: changes ParseStream for native format uses reader instead of http.Request updates app/vmagent and app/vmagent method usage * app/vmctl: add verify-block subcommand it allows to check exported from VictoriaMetrics data block in native format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362 Update app/vmctl/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-04-05 17:46:36 +03:00
Aliaksandr Valialkin	fca0cb8156	lib/workingsetcache: reduce the expiration duration from 20 minutes to 10 minutes This should reduce memory usage for the cache under high churn rate	2022-04-05 17:08:43 +03:00
Aliaksandr Valialkin	8752cce157	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage This reduces the max memory usage for vminsert and vmstorage under heavy ingestion rate by up to 50% on production workload	2022-04-05 15:39:58 +03:00
Aliaksandr Valialkin	a8337c7170	vendor: `make vendor-update`	2022-04-04 13:08:07 +03:00
Roman Khavronenko	ce1629b70a	vmalert: add flag for disabling long-lived connections (keepalive) (#2395 ) The new flag `datasource.disableKeepAlive` allows disabling keepalive connections. This may be useful if there are multiple datasource replicas (e.g. vmselects) behind the HTTP balancer to avoid uneven load spread because of long-lived connections. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-04 13:08:06 +03:00
Aliaksandr Valialkin	816085a652	docs/CHANGELOG.md: document `70bb0d2708`	2022-04-04 13:02:41 +03:00
Aliaksandr Valialkin	676c1f4fd7	docs/CHANGELOG.md: document 173073364e1bb1e0259ddc873dbd96ce62b07543	2022-04-04 12:56:11 +03:00
Artem Navoiev	23614d752a	Update release process. Actualize helm-charts release process. Add re… (#2397 ) * Update release process. Actualize helm-charts release process. Add release guide for ansible playbooks * add step to Ansible steps	2022-04-04 12:56:11 +03:00
Aliaksandr Valialkin	feff0ef286	app/vmselect: `make vmui-update`	2022-04-01 12:55:53 +03:00
dependabot[bot]	bb3ec058cb	build(deps): bump react-router-dom in /app/vmui/packages/vmui (#2394 ) Bumps [react-router-dom](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom) from 6.2.2 to 6.3.0. - [Release notes](https://github.com/remix-run/react-router/releases) - [Commits](https://github.com/remix-run/react-router/commits/v6.3.0/packages/react-router-dom) --- updated-dependencies: - dependency-name: react-router-dom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-01 12:54:10 +03:00
dependabot[bot]	86afc4b4ad	build(deps): bump @testing-library/react in /app/vmui/packages/vmui (#2392 ) Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 12.1.4 to 13.0.0. - [Release notes](https://github.com/testing-library/react-testing-library/releases) - [Changelog](https://github.com/testing-library/react-testing-library/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/react-testing-library/compare/v12.1.4...v13.0.0) --- updated-dependencies: - dependency-name: "@testing-library/react" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-01 12:53:15 +03:00
dependabot[bot]	f478429f8f	build(deps): bump @types/react in /app/vmui/packages/vmui (#2375 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.41 to 17.0.43. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-01 12:52:21 +03:00
dependabot[bot]	d6f9eaf3a5	build(deps): bump @testing-library/user-event in /app/vmui/packages/vmui (#2393 ) Bumps [@testing-library/user-event](https://github.com/testing-library/user-event) from 13.5.0 to 14.0.4. - [Release notes](https://github.com/testing-library/user-event/releases) - [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/user-event/compare/v13.5.0...v14.0.4) --- updated-dependencies: - dependency-name: "@testing-library/user-event" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-04-01 12:51:52 +03:00
Yury Molodov	b2f0acaca4	vmui: grid support for predefined panels (#2386 ) * update packages * feat: add setting width for predefined panels * docs: update doc by predefined dashboards * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-01 12:49:24 +03:00
Ross Dougherty	4df09074d3	fix object selectors link (#2391 ) * fix object selectors link * update kustomize url	2022-04-01 12:41:28 +03:00
Dima Lazerka	e756eae06d	Fix typo "vmanomapy"	2022-04-01 12:26:53 +03:00
Aliaksandr Valialkin	157555b022	docs/CHANGELOG.md: document a57e3807537914396ee3eb378648a464fa9e1b97	2022-04-01 12:25:54 +03:00
Aliaksandr Valialkin	151a00189e	docs/CHANGELOG.md: document `0989649ad0`	2022-04-01 12:03:41 +03:00
Yurii Kravets	bd0068254d	url-examples (#2389 )	2022-04-01 12:03:41 +03:00
Roman Khavronenko	7aa9d0f5f6	vmalert: protect executor's field from concurrent access (#2387 ) Executor recently gain field for storing previously sent series. Since the same executor object can be used in multiple goroutines, the access to this field should be serialized. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-01 12:03:41 +03:00
Roman Khavronenko	ab10178c85	Vmalert compliance 2 (#2340 ) * vmalert: split alert's `Start` field into `ActiveAt` and `Start` The `ActiveAt` field identifies when alert becomes active for rules with `for > 0`. Previously, this value was stored in field `Start`. The field `Start` now identifies the moment alert became `FIRING`. The split is needed in order to distinguish these two moments in the API responses for alerts. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: support specific moment of time for rules evaluation The Querier interface was extended to accept a new argument used as a timestamp at which evaluation should be made. It is needed to align rules execution time within the group. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: mark disappeared series as stale Series generated by alerting rules, which were sent to remote write now will be marked as stale if they will disappear on the next evaluation. This would make ALERTS and ALERTS_FOR_TIME series more precise. Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: evaluate rules at fixed timestamp Before, time at which rules were evaluated was calculated right before rule execution. The change makes sure that timestamp is calculated only once per evalution round and all rules are using the same timestamp. It also updates the logic of resending of already resolved alert notification. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: allow overridin `alertname` label value if it is present in response Previously, `alertname` was always equal to the Alerting Rule name. Now, its value can be overriden if series in response containt the different value for this label. The change is needed for improving compatibility with Prometheus. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: align rules evaluation in time Now, evaluation timestamp for rules evaluates as if there was no delay in rules evaluation. It means, that rules will be evaluated at fixed timestamps+group_interval. This way provides more consistent evaluation results and improves compatibility with Prometheus, Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: add metric for missed iterations New metric `vmalert_iteration_missed_total` will show whether rules evaluation round was missed. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: reduce delay before the initial rule evaluation in group Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: rollback alertname override According to the spec: ``` The alert name from the alerting rule (HighRequestLatency from the example above) MUST be added to the labels of the alert with the label name as alertname. It MUST override any existing alertname label. ``` https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#step-3 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: throw err immediately on dedup detection ``` The execution of an alerting rule MUST error out immediately and MUST NOT send any alerts or add samples to samples receiver if there is more than one alert with the same labels ``` https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#step-4 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: cleanup Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: use strings builder to reduce allocs Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-01 12:03:41 +03:00
Denys Holius	26cc40ab00	Update alpine linux base image to the latest v3.15.3 (#2384 ) Updated alpine linux base image to the latest v3.15.3 which has fix for [CVE-2018-25032](https://security.alpinelinux.org/vuln/CVE-2018-25032). See https://alpinelinux.org/posts/Alpine-3.12.11-3.13.9-3.14.5-3.15.3-released.html	2022-04-01 12:03:41 +03:00
Roman Khavronenko	d907e0d9f0	docs: fix typo in vmalert's API (#2380 ) The API handler was changed in 1.75 but docs still contain the old address. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2366 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-04-01 12:03:41 +03:00
Roman Khavronenko	5b367ad594	docs: update `increase_prometheus` desc (#2381 ) Remove note that this func is supported by PromQL.	2022-03-28 12:29:41 +03:00
Roman Khavronenko	44fa461e77	docs: escape chars for /label/values endpoint (#2379 ) Without escaping the part wrapped with `<` `>` chars won't be rendered properly.	2022-03-28 12:29:41 +03:00
Aliaksandr Valialkin	7c81ce296d	docs/CHANGELOG.md: cut v1.75.1	2022-03-28 12:28:27 +03:00
Aliaksandr Valialkin	20cecf32c0	vendor: `make vendor-update`	2022-03-26 13:08:29 +02:00
Yury Molodov	7cbf19812d	vmui: predefined panels (#2243 ) * feat: add basic components for predefined dashboards * fix: change display alert * feat: add autosize and unit for axes * feat: add component for CircularProgress * feat: change layout for predefined dashboards * feat: add override step for predefined panels * feat: add override step for predefined panels * feat: change yaxis limits for predefined panels * fix: rename flag for hide legend * feat: add formatted panel description * feat: add README.md for dashboard setup * feat: validate dashboard settings * feat: add unit for y-ticks * fix: correct display error for dashboards * fix: disable auto refresh after route change * update package-lock.json * fix: add basename for BrowserRouter * fix: add dynamic basename for routing * update packages * feat: add a pre-defined dashboard "per-job resource usage" * feat: display unit in the hover-tooltip * fix: change routing and home layout * fix: change axis width calc * updated packages * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 13:05:21 +02:00
dependabot[bot]	56c32011c3	build(deps): bump node-forge in /app/vmui/packages/vmui (#2371 ) Bumps [node-forge](https://github.com/digitalbazaar/forge) from 1.2.1 to 1.3.0. - [Release notes](https://github.com/digitalbazaar/forge/releases) - [Changelog](https://github.com/digitalbazaar/forge/blob/main/CHANGELOG.md) - [Commits](https://github.com/digitalbazaar/forge/compare/v1.2.1...v1.3.0) --- updated-dependencies: - dependency-name: node-forge dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-26 12:58:00 +02:00
Nikolay	4cf6219e07	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293 ) * lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache It should decrease memory usage for regexp caching with storing cacheEntry by pointer - golang map should be able to effectivly shrink it's size original issue with this case - unexpected map grows and storage OOM Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Adds missing metrics for regexp cache and regexpPrefixes cache * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 12:57:27 +02:00
Denys Holius	fb0fb41ea9	Update alpine linux base image to the latest v3.15.2 Update alpine linux base image to the latest v3.15.2 which has fix for CVE-2022-0778. See https://alpinelinux.org/posts/Alpine-3.15.2-released.html	2022-03-26 11:30:36 +02:00
Aliaksandr Valialkin	b843f0e229	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:28:14 +02:00
Dima Lazerka	6e8e385375	VMAnomaly docs fixes (#2361 ) * Added docs for vmanomaly * Add example images * Stylistic fixes * Move images to root * Update docs/vmanomaly.md * Update docs/vmanomaly.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * Squeeze vmanomaly after vmbackupmanager before Case Studies Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-03-25 12:09:05 +02:00
Aliaksandr Valialkin	b7b06fb50e	Revert "docs/CHANGELOG.md: mention a bugfix in Graphite render API in v1.75.0" This reverts commit `1e18c1c1ae`. The bugfix has been alread mentioned in the commit `9290605891`	2022-03-24 19:21:15 +02:00
Aliaksandr Valialkin	1e18c1c1ae	docs/CHANGELOG.md: mention a bugfix in Graphite render API in v1.75.0	2022-03-24 19:20:42 +02:00
Denys Holius	259ea7ee39	Update golangci version to latest v1.45.1 (#2360 )	2022-03-24 19:18:02 +02:00
Dima Lazerka	50bc179dce	Add vmanomaly docs section (#2356 ) * Added docs for vmanomaly * Add example images * Stylistic fixes * Move images to root * Update docs/vmanomaly.md * Update docs/vmanomaly.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-03-24 11:57:07 +02:00
Yurii Kravets	b8d9c071e9	Update url-examples (#2358 ) * Update url-examples Add federate example * Update docs/url-examples.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-24 11:57:06 +02:00
Aliaksandr Valialkin	c3d7522c19	docs: a follow-up after `76a477c609`	2022-03-23 19:39:58 +02:00
Yurii Kravets	5f41e22cb1	Update Single-server-VictoriaMetrics.md (#2357 ) * Update Single-server-VictoriaMetrics.md Adding /federate link * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-23 19:39:57 +02:00
Yurii Kravets	a8b55103aa	Update url-examples.md (#2297 ) * Update url-examples.md +additional examples * Update * Update url-examples Some changes requested by Roman. * Update url-examples.md * Update url-example * Update url-examples Additional info and marking for /labels part * Update url-example Added example with complex query which needs encoding: How to execute the query similar to - sum(increase(foo{status="bar"}[5m])) by (status) * Update url-samples Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-22 15:12:36 +02:00
Aliaksandr Valialkin	ba7cfd7b25	app: sync Markdown changes from `a8de1ab000`	2022-03-22 14:12:03 +02:00
Aliaksandr Valialkin	8f678411cc	docs/Cluster-VictoriaMetrics.md: clarify mTLS protection docs	2022-03-22 14:01:06 +02:00
Arash Hatami	2e01691d5d	A good change for MD files (#2353 ) * Lint YAML * Remove extra comment * Fix command problem * Format MD files * Format & fix problem of MD files for docs * Another fix for MD files	2022-03-22 14:01:04 +02:00
Aliaksandr Valialkin	3ed2f67876	vendor: `make vendor-update`	2022-03-21 17:02:42 +02:00
dependabot[bot]	6be02c82a0	build(deps): bump @types/react in /app/vmui/packages/vmui (#2346 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.40 to 17.0.41. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-21 15:45:27 +02:00
dependabot[bot]	30471f88a6	build(deps): bump @types/react-dom in /app/vmui/packages/vmui (#2347 ) Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 17.0.13 to 17.0.14. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom) --- updated-dependencies: - dependency-name: "@types/react-dom" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-21 15:45:26 +02:00
Roman Khavronenko	bb594b34b8	docs: update release notes (#2349 ) Warn about memory issue introduced in releases 1.73 - 1.74	2022-03-21 15:41:25 +02:00
Aliaksandr Valialkin	84b3234f3d	docs/CHANGELOG.md: document `a1e17e91f8`	2022-03-21 15:37:03 +02:00
hagen1778	7c7436b584	docs: add update note to v1.75.0 release note Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-03-21 15:37:01 +02:00
Aliaksandr Valialkin	7e67d61fdf	docs/CHANGELOG.md: cut v1.75.0	2022-03-18 19:54:13 +02:00
Aliaksandr Valialkin	85695faa94	app/vmselect/bufferedwriter: suppress trivial network errors, which can be generated by remote side These errors include `broken pipe` and `reset by peer`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2334	2022-03-18 19:27:33 +02:00
Aliaksandr Valialkin	9d91ad7124	app/vmagent/remotewrite: prevent from infinite recursion panic when pushing a time series with big number of samples to remote storage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2335	2022-03-18 19:07:27 +02:00
Aliaksandr Valialkin	ab966f8a7a	docs: document `20bb5e703c`	2022-03-18 18:42:09 +02:00
Diego Circelli	43dfa421c3	Token authorization scheme (#2288 )	2022-03-18 18:42:07 +02:00
Aliaksandr Valialkin	a8a4581c37	lib/blockcache: properly remove references to deleted parts Previously references to deleted parts may remain active as cache.m keys. This could prevent from proper memory de-allocation. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This is a follow-up for `88605a7ea2`	2022-03-18 17:07:54 +02:00
Aliaksandr Valialkin	e35c9124b7	lib/storage: reduce the interval for checking for free disk space from 30 seconds to 1 second This should reduce the probability of out of disk space panics when -storage.minFreeDiskSpaceBytes is set to low values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2305	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	7c92aaeaa4	lib/blockcache: properly release memory occupied by deleted entries Proviously the deleted entries could remain referenced via lastAccessHeap for long time. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	a6d65fc824	lib/storage: typo fix after `e7831ae154`	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	95da5e52c4	app/vmselect/vmui: `make vmui-update`	2022-03-18 13:18:57 +02:00
dependabot[bot]	2423edf4ec	build(deps): bump @testing-library/react in /app/vmui/packages/vmui (#2333 ) Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 12.1.3 to 12.1.4. - [Release notes](https://github.com/testing-library/react-testing-library/releases) - [Changelog](https://github.com/testing-library/react-testing-library/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/react-testing-library/compare/v12.1.3...v12.1.4) --- updated-dependencies: - dependency-name: "@testing-library/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 13:13:28 +02:00
dependabot[bot]	6d29a775cf	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2336 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.5.0 to 5.5.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.5.1/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 13:13:28 +02:00
dependabot[bot]	e6ad999311	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2337 ) Bumps [@mui/icons-material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-icons-material) from 5.4.4 to 5.5.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.5.1/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 13:13:28 +02:00
dependabot[bot]	08a7dddac4	build(deps-dev): bump @typescript-eslint/parser (#2338 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.13.0 to 5.15.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.15.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 13:13:28 +02:00
dependabot[bot]	d8ed4980db	build(deps): bump @types/react in /app/vmui/packages/vmui (#2339 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.39 to 17.0.40. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 13:13:28 +02:00
Aliaksandr Valialkin	85a4b805e1	docs/CHANGELOG.md: document `e5868b9c29` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2255	2022-03-18 13:08:56 +02:00
jduncan0000	e7831ae154	Fix for issue #2255 - matchTagFilters for positive empty-match filters (#2304 ) * fix for issue 2255 - matchTagFilters for positive empty-match filters * add example to comments * formatting * add test for positive empty match * formatting	2022-03-18 13:08:54 +02:00
dependabot[bot]	7d759829dd	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2330 ) Bumps [@mui/lab](https://github.com/mui/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.71 to 5.0.0-alpha.73. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:32:37 +02:00
dependabot[bot]	3f29cd0682	build(deps): bump typescript in /app/vmui/packages/vmui (#2286 ) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.5.5 to 4.6.2. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](https://github.com/Microsoft/TypeScript/compare/v4.5.5...v4.6.2) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:31:25 +02:00
dependabot[bot]	85b560488c	build(deps-dev): bump eslint-plugin-react in /app/vmui/packages/vmui (#2313 ) Bumps [eslint-plugin-react](https://github.com/yannickcr/eslint-plugin-react) from 7.29.2 to 7.29.4. - [Release notes](https://github.com/yannickcr/eslint-plugin-react/releases) - [Changelog](https://github.com/yannickcr/eslint-plugin-react/blob/master/CHANGELOG.md) - [Commits](https://github.com/yannickcr/eslint-plugin-react/compare/v7.29.2...v7.29.4) --- updated-dependencies: - dependency-name: eslint-plugin-react dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:29:57 +02:00
dependabot[bot]	2e3d849637	build(deps): bump dayjs from 1.10.8 to 1.11.0 in /app/vmui/packages/vmui (#2332 ) Bumps [dayjs](https://github.com/iamkun/dayjs) from 1.10.8 to 1.11.0. - [Release notes](https://github.com/iamkun/dayjs/releases) - [Changelog](https://github.com/iamkun/dayjs/blob/v1.11.0/CHANGELOG.md) - [Commits](https://github.com/iamkun/dayjs/compare/v1.10.8...v1.11.0) --- updated-dependencies: - dependency-name: dayjs dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:29:57 +02:00
dependabot[bot]	61060352f2	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#2331 ) Bumps [@mui/styles](https://github.com/mui/material-ui/tree/HEAD/packages/mui-styles) from 5.4.4 to 5.5.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.5.1/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:29:57 +02:00
dependabot[bot]	235e4cda46	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2329 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.13.0 to 5.15.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.15.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:29:57 +02:00
dependabot[bot]	fd010cd8bf	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2310 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.4.4 to 5.5.0. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.5.0/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-18 12:29:57 +02:00
Aliaksandr Valialkin	3b7aefade2	docs/vmalert.md: sync after `11ae1ae924`	2022-03-17 20:18:07 +02:00
Aliaksandr Valialkin	e1f70af14e	docs/CHANGELOG.md: document `11ae1ae924`	2022-03-17 20:09:18 +02:00
Dmytro Kozlov	2f350c200a	Added resendDelay for alerts (#2296 ) * vmalert: add support of `resendDelay` flag for alerts Co-authored-by: dmitryk-dk <dmitry.kozlov@brightlocal.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>	2022-03-17 20:09:18 +02:00
Aliaksandr Valialkin	f0e96a84db	docs: document the addition of mTLS communication between cluster components	2022-03-17 20:00:14 +02:00
Aliaksandr Valialkin	698458b742	lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig	2022-03-17 19:46:20 +02:00
Aliaksandr Valialkin	9290605891	docs/CHANGELOG.md: document c1d07e7c52f0a2ab892921b0639cd42677aa33a8	2022-03-16 14:25:38 +02:00
Aliaksandr Valialkin	0e8bd3b4cf	deployment/docker: upgrade Go builder for production builds from v1.17.7 to v1.18.0 See https://tip.golang.org/doc/go1.18	2022-03-16 14:08:52 +02:00
Aliaksandr Valialkin	f31f097de0	vendor: `make vendor-update`	2022-03-16 13:55:25 +02:00
Aliaksandr Valialkin	c175afbe02	docs/CHANGELOG.md: document changes from `fb6eab03a2`	2022-03-16 13:28:29 +02:00
Aliaksandr Valialkin	191977b324	lib/storage: trashing -> thrashing typo in docs This is a follow-up for `918ed5cb32`	2022-03-16 13:28:29 +02:00
Aliaksandr Valialkin	5b53373154	docs/CHANGELOG.md: document `565bd08c43` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1824	2022-03-16 13:28:29 +02:00
Aliaksandr Valialkin	bf131a4990	docs: `make docs-sync`	2022-03-16 13:28:29 +02:00
Roman Khavronenko	35bf5bf688	Vmalert compliance improvements (#2320 ) * vmalert: add support for `sortByLabel` template function * vmalert: update API according to Prometheus conformance program The changes to the API, field names and URL path has been made according to the Prometheus specification for `alert_generator` https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md * vmalert: fix the timestamp of the evaluated rules The timestamp used for alert's `EndsAt` was calculated before sending the notification. While the correct way is to use the timestamp taken right before rules evaluation. * vmalert: add `-datasource.queryTimeAlignment` flag The flag is supposed to provide ability to disable `time` param alignment when executing rules. By default, this flag is enabled, so it remains backward compatible. The flag was introduced to achieve better compatibility with Prometheus behaviour according to https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-03-16 13:22:26 +02:00
Vic (Shihang) Li	9767fcd837	fix: change thrashing typo (#2317 )	2022-03-16 13:05:55 +02:00
hagen1778	fd0f521bb9	docs: add update details for some releases Some of the releases could negatively affect performance for a limited period of time due to some changes in core. Update details are meant to warn users about expected changes in peformance after the update. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-03-16 13:03:36 +02:00
Roman Khavronenko	ec48d022df	docs: fix broken links (#2303 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-03-16 13:01:46 +02:00
Dmytro Kozlov	7f0212f964	Issue-1824: added flags and different auth types support (#2287 ) * vmalert/notifier: added flags and different auth types support Co-authored-by: hagen1778 <roman@victoriametrics.com>	2022-03-16 12:52:01 +02:00
Ted Robertson	d4f2dd2a71	Clarify lack of support for InfluxQL (#2273 )	2022-03-16 12:25:09 +02:00
Bastien Dronneau	da722e462e	docs(vmalert): typo in path (#2278 )	2022-03-16 12:24:12 +02:00
Aliaksandr Valialkin	e7a0b4e095	docs/CHANGELOG.md: cut v1.74.0	2022-03-03 19:30:53 +02:00
Yury Molodov	235ef947ed	vmui: update packages (#2264 ) * update package.json * update package-lock.json * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-03 18:50:44 +02:00
Aliaksandr Valialkin	419f109f88	docs: `make docs-sync` after `702aa4948b`	2022-03-03 18:37:07 +02:00
Dan Dascalescu	e15b27b792	docs: clarify Retention, improve English (#2266 )	2022-03-03 18:37:07 +02:00
nemobis	cd522dee14	Fix typo, sentence flow in operator description (#2251 )	2022-03-03 18:37:07 +02:00
dependabot[bot]	6db97a49a1	build(deps): bump @types/node in /app/vmui/packages/vmui (#2262 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.19 to 17.0.21. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-03 18:37:07 +02:00
dependabot[bot]	eed191a25b	build(deps-dev): bump eslint-plugin-react in /app/vmui/packages/vmui (#2261 ) Bumps [eslint-plugin-react](https://github.com/yannickcr/eslint-plugin-react) from 7.28.0 to 7.29.2. - [Release notes](https://github.com/yannickcr/eslint-plugin-react/releases) - [Changelog](https://github.com/yannickcr/eslint-plugin-react/blob/master/CHANGELOG.md) - [Commits](https://github.com/yannickcr/eslint-plugin-react/compare/v7.28.0...v7.29.2) --- updated-dependencies: - dependency-name: eslint-plugin-react dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-03 18:37:07 +02:00
dependabot[bot]	8002f728a6	build(deps): bump @types/jest in /app/vmui/packages/vmui (#2260 ) Bumps [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest) from 27.4.0 to 27.4.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest) --- updated-dependencies: - dependency-name: "@types/jest" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-03-03 18:37:07 +02:00
Aliaksandr Valialkin	e2eaf2a681	vendor: `make vendor-update`	2022-03-03 18:15:12 +02:00
Aliaksandr Valialkin	0b5ec1780d	docs/CHANGELOG.md: document performance improvements when registering new time series	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	3f999b11f7	lib/mergeset: remove aux buffers from inmemoryPart This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf68da79e	lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock This should improve performance when registering new time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf4f7bf21	lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	f4e466955d	lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart, so sync.Pool usage should reduce overall memory usage and improve performance because of better locality of reference when marshaling inmemoryBlock to inmemoryPart. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	b47f18f555	lib/{mergeset,storage}: tune compression levels for small blocks This should reduce CPU usage spent on compression	2022-02-25 15:34:13 +02:00
Nikolay	d5ba1249f8	fixes incorrect step for calculation for MovingWindow functions (#283 ) * fixes incorrect step for calculation for MovingWindow functions https://victoriametrics.zendesk.com/agent/tickets/99 * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-25 13:53:48 +02:00
Aliaksandr Valialkin	1bde8b4e22	docs/CHANGELOG.md: document cfc6c14dc48ae9dd35e65f1a6e5c7af8ccb9f029	2022-02-25 13:53:09 +02:00
Aliaksandr Valialkin	28b610db07	lib/storage: document why job-like and instance-like labels must be stored at mn.Tags[0] and mn.Tags[1] Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2244	2022-02-25 13:21:53 +02:00
Aliaksandr Valialkin	d1881fa582	lib/storage: add a comment to indexSearch.containsTimeRange() on why it allows false positives Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2239	2022-02-24 12:48:33 +02:00
Aliaksandr Valialkin	02a922b53f	lib/storage: properly handle series selector matching multiple metric names plus a negative filter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238 This is a follow-up for `00cbb099b6`	2022-02-24 12:11:53 +02:00
Aliaksandr Valialkin	1967b9c211	lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort() There is no need to sort the underlying data according to sorted items there. This should reduce cpu usage when registering new time series in `indexdb`. Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245	2022-02-24 11:19:44 +02:00
Aliaksandr Valialkin	99ee1c13c0	app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues	2022-02-24 04:03:28 +02:00
Aliaksandr Valialkin	2431c9cf81	lib/promrelabel: add support for conditional relabeling via `if` filter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998	2022-02-24 02:35:13 +02:00
Aliaksandr Valialkin	6fd85117ac	lib/workingsetcache: do not rotate cache if it is in `whole` state This should reduce the maximum memory usage for the cache in `whole` state	2022-02-23 22:55:10 +02:00
Nikolay	54221297d1	fixes jwt token parse with correct base64Url decoding (#281 ) * fixes jwt token parse with correct base64Url decoding it must be applied according to jwt RFC that requires token to be URL safe added slow path for decoding tokens with std base64 decoding adds error logging for vmgateway * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-23 13:57:54 +02:00
Aliaksandr Valialkin	244c23ea2c	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes This should reduce memory usage under high time series churn rate	2022-02-23 13:42:27 +02:00
Aliaksandr Valialkin	cb100ce2af	docs/CHANGELOG.md: cut v1.73.1	2022-02-22 21:11:55 +02:00
Aliaksandr Valialkin	701eb0495b	app/vmselect: `make vmui-update`	2022-02-22 21:02:20 +02:00
dependabot[bot]	9faf3ee2ca	build(deps): bump @types/node in /app/vmui/packages/vmui (#280 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.18 to 17.0.19. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 21:02:20 +02:00
dependabot[bot]	e39bba870b	build(deps): bump @emotion/styled in /app/vmui/packages/vmui (#279 ) Bumps [@emotion/styled](https://github.com/emotion-js/emotion) from 11.6.0 to 11.8.1. - [Release notes](https://github.com/emotion-js/emotion/releases) - [Changelog](https://github.com/emotion-js/emotion/blob/main/CHANGELOG.md) - [Commits](https://github.com/emotion-js/emotion/compare/@emotion/styled@11.6.0...@emotion/styled@11.8.1) --- updated-dependencies: - dependency-name: "@emotion/styled" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 21:02:20 +02:00
dependabot[bot]	bd6c5ae35e	build(deps): bump @mui/material in /app/vmui/packages/vmui (#278 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.4.2 to 5.4.3. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.3/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 21:02:20 +02:00
dependabot[bot]	cfe373a92b	build(deps-dev): bump @typescript-eslint/eslint-plugin (#276 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.11.0 to 5.12.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.12.1/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 21:02:20 +02:00
Aliaksandr Valialkin	d51c838904	vendor: `make vendor-update`	2022-02-22 20:56:33 +02:00
dependabot[bot]	cf4f7c902b	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#277 ) Bumps [@mui/lab](https://github.com/mui/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.68 to 5.0.0-alpha.70. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 20:45:50 +02:00
dependabot[bot]	333243c539	build(deps-dev): bump @typescript-eslint/parser (#275 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.11.0 to 5.12.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.12.1/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 20:45:50 +02:00
dependabot[bot]	966b3319d7	build(deps): bump @testing-library/react in /app/vmui/packages/vmui (#274 ) Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 12.1.2 to 12.1.3. - [Release notes](https://github.com/testing-library/react-testing-library/releases) - [Changelog](https://github.com/testing-library/react-testing-library/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/react-testing-library/compare/v12.1.2...v12.1.3) --- updated-dependencies: - dependency-name: "@testing-library/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 20:45:50 +02:00
dependabot[bot]	be6b428d51	build(deps): bump preact in /app/vmui/packages/vmui (#272 ) Bumps [preact](https://github.com/preactjs/preact) from 10.6.5 to 10.6.6. - [Release notes](https://github.com/preactjs/preact/releases) - [Commits](https://github.com/preactjs/preact/compare/10.6.5...10.6.6) --- updated-dependencies: - dependency-name: preact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 20:45:50 +02:00
dependabot[bot]	ccded15fdb	build(deps): bump @date-io/dayjs in /app/vmui/packages/vmui (#263 ) Bumps [@date-io/dayjs](https://github.com/dmtrKovalenko/date-io) from 2.11.0 to 2.13.1. - [Release notes](https://github.com/dmtrKovalenko/date-io/releases) - [Commits](https://github.com/dmtrKovalenko/date-io/compare/v2.11.0...v2.13.1) --- updated-dependencies: - dependency-name: "@date-io/dayjs" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-22 20:45:50 +02:00
Aliaksandr Valialkin	1ff42a0080	docs/CHANGELOG.md: link to the feature request for X-Influxdb-Version response header Follow-up for `71ef3155c8` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2209	2022-02-22 20:34:03 +02:00
Yurii Kravets	538114cd7c	Update URL examples (#2207 ) added div with copy function	2022-02-22 20:09:27 +02:00
Denys Holius	fc6f71fba8	Added minimal supported version of AlertManager (#2237 ) * added minimal supported version of supported AlertManager * docs: `make docs-sync` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-22 20:09:26 +02:00
Aliaksandr Valialkin	89ead3daca	app/vmselect/netstorage: report vmstorage errors to vmselect clients even if partial responses are allowed If a vmstorage is reachable and returns an application-level error to vmselect, then such error must be returned to the caller even if partial responses are allowed, since it usually means cluster mis-configuration. Partial responses may be returned only if some vmstorage nodes are temporarily unavailable. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1941 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/678	2022-02-21 21:17:05 +02:00
Aliaksandr Valialkin	fcaa0c5202	lib/storage: optimize `/api/v1/status/tsdb` call by skipping all the artificially created tag entries at once This is a follow-up for `b71be42d90`	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	986c12487a	lib/mergeset: typo fix after `b6ed9afd6d`	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	d4df32cc6b	lib/blockcache: evict entries from the cache in LRU order This should improve hit rate for smaller caches	2022-02-21 19:00:04 +02:00
Yury Molodov	cdd632985f	vmui: change logic autocomplete (#2196 ) * fix: change autocomplete display logic * fix: change z-index for labels of input fields * fix: change autocomplete display logic * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:44:54 +02:00
dependabot[bot]	4211f85e52	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#2225 ) Bumps [@mui/styles](https://github.com/mui/material-ui/tree/HEAD/packages/mui-styles) from 5.4.1 to 5.4.2. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.2/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 15:35:01 +02:00
Roman Khavronenko	5a4b16794d	Consul SD - update services on the watcher's start (#2202 ) * lib/discovery/consul: update services on the watcher's start Previously, watcher's start was only initing goroutines for discovery but not waiting for the first iteration to end. It means first Consul discovery wasn't returning discovered targets until the next iteration. The change makes the watcher's start blocking until we get first discovery iteration done and all registries updated. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: remove workarounds for consul SD Now when consul SD lib properly updates services on the first start, we don't need workarounds in vmalert. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/discovery/consul: update after review Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:33:33 +02:00
dependabot[bot]	2ec6d35a61	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2226 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.4.1 to 5.4.2. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.2/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 14:14:07 +02:00
dependabot[bot]	8bd73aac2b	build(deps-dev): bump react-app-rewired in /app/vmui/packages/vmui (#2227 ) Bumps [react-app-rewired](https://github.com/timarney/react-app-rewired) from 2.1.11 to 2.2.1. - [Release notes](https://github.com/timarney/react-app-rewired/releases) - [Commits](https://github.com/timarney/react-app-rewired/compare/v2.1.11...v2.2.1) --- updated-dependencies: - dependency-name: react-app-rewired dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 14:14:07 +02:00
dependabot[bot]	77f50e8bd1	build(deps): bump @types/node in /app/vmui/packages/vmui (#2228 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.17 to 17.0.18. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 14:14:07 +02:00
dependabot[bot]	ace196da8d	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2229 ) Bumps [@mui/icons-material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-icons-material) from 5.4.1 to 5.4.2. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.2/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 14:14:07 +02:00
dependabot[bot]	ba53a69f4a	build(deps): bump nanoid from 3.1.30 to 3.3.1 in /app/vmui/packages/vmui (#2232 ) Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.30 to 3.3.1. - [Release notes](https://github.com/ai/nanoid/releases) - [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md) - [Commits](https://github.com/ai/nanoid/compare/3.1.30...3.3.1) --- updated-dependencies: - dependency-name: nanoid dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-21 14:14:07 +02:00
Aliaksandr Valialkin	4a658cf955	app/vmagent: add `-influxDBLabel` in the same way as for app/vminsert This is a follow-up for `3d19fa6932`	2022-02-21 14:09:40 +02:00
Aliaksandr Valialkin	28edbd9abb	vendor: `make vendor-update`	2022-02-21 13:59:20 +02:00
Roman Khavronenko	bd7837d524	lib: allow to configure cache size by type (#2206 ) * lib: allow to configure cache size by type https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 13:55:51 +02:00
Pablo Ley	4606d00d7d	Added missing inlineScrapeConfig field to the VMAgentSpec docs	2022-02-21 13:55:50 +02:00
Denys Holius	fe735fcca9	Added missed runbook for udpating k8s VM Cluster in DO (#2219 ) * added missed runbook for udpating k8s VM Cluster in DO * Update deployment/marketplace/digitialocean/one-click-droplet/RELEASE_GUIDE.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-18 16:29:53 +02:00
Aliaksandr Valialkin	47b93a1337	docs: `make docs-sync`	2022-02-18 12:26:35 +02:00
Roman Khavronenko	f59685e5ff	docs: make profiling commands more copy-friendly (#2213 ) * docs: make profiling commands more copy-friendly The change adds `copy text` snippet to code examples and replaces hostname placeholders with `0.0.0.0`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-18 12:26:35 +02:00
Denys Holius	bf0b31e7bb	Bump golangci version (#2214 )	2022-02-18 12:26:35 +02:00
Aliaksandr Valialkin	5260d7a954	lib/storage: typo fix after `c3affb0c4f`	2022-02-17 12:56:33 +02:00
Aliaksandr Valialkin	ce91456982	app/vminsert: add `X-Influxdb-Version` response header for InfluxDB API requests This is needed for some clients, which expect this header. See https://github.com/ntop/ntopng/issues/5449#issuecomment-1005347597	2022-02-17 12:49:24 +02:00
Aliaksandr Valialkin	d9bdb42219	lib/storage: simplify code for searching for label values This is a follow-up after `9dd191b27c`	2022-02-17 12:39:14 +02:00
Aliaksandr Valialkin	8aaeffa7b4	docs: document `3d19fa6932`	2022-02-16 23:30:52 +02:00
Eugene	8e1c78d5ab	Add -influxDBLabel parameter (#2203 )	2022-02-16 23:25:01 +02:00
Aliaksandr Valialkin	2ebc3d21c3	lib/storage: properly skip composite tag entries when searching for tag names or tag values This is a follow-up for `b71be42d90` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 23:02:18 +02:00
Aliaksandr Valialkin	3107224306	lib/blockcache: fix TestCache by ensuring that the cache size can be divided by the number of cache shards Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2204	2022-02-16 18:48:09 +02:00
Aliaksandr Valialkin	63bc89dd81	lib/storage: document why tsid cache is reset before saving it to disk Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2205	2022-02-16 18:37:29 +02:00
Aliaksandr Valialkin	ee066aa0d5	lib/storage: use binary search instead of full scan for skipping artificial tags when searching for tag names or tag values This should improve performance for /api/v1/labels and /api/v1/label/<label_name>/values See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 18:17:27 +02:00
Aliaksandr Valialkin	f227712d7d	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics doesnt check DD_API_KEY when accepting data via DataDog protocol	2022-02-16 14:40:01 +02:00
匠心零度	cbba78e091	Correct URL:How to delete time series (#2201 ) Co-authored-by: lirenzuo <lirenzuo@shein.com>	2022-02-16 14:39:59 +02:00
Roman Khavronenko	3569352fe0	dashboards: update the threshold for slow inserts % on the dashboard (#2198 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-15 21:57:21 +02:00
Aliaksandr Valialkin	eaf29a5e8e	docs/url-examples.md: formatting fixes	2022-02-15 21:25:09 +02:00
Aliaksandr Valialkin	bb113efeb4	docs/CHANGELOG.md: document `2efa46a11c`	2022-02-15 21:13:32 +02:00
Denys Holius	7e73fafc6f	Update revisions of dashboards (#2199 ) * update revision of vmcluster for operator guide * update revision of vmcluster for vmcluster guide	2022-02-15 21:13:32 +02:00
hagen1778	11345c3fd1	vmalert: support `$externalLabels` and `$externalURL` in templates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2193 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-15 21:13:32 +02:00
Yurii Kravets	759ea64b85	Create url-examples.md	2022-02-15 21:13:32 +02:00
Aliaksandr Valialkin	5aa3c896a8	docs/Single-server-VictoriaMetrics.md: clarify that the data is imported into VictoriaMetrics at `how to import data` section	2022-02-15 21:07:04 +02:00
Aliaksandr Valialkin	7e8596edb8	docs: update `-help` output for VictoriaMetrics components	2022-02-15 21:02:36 +02:00
Oct	c7c45bf842	docs: correct address in the docs (#2189 ) * docs/FAQ.md:fix 404 url * app/vmui/packages/vmui/README.md:rm 404 url	2022-02-15 12:59:50 +02:00
Roman Khavronenko	8cbcc560b9	vmagent: fix js error on CollapseAll/ExpandAll buttons click (#2192 ) * vmagent: fix js error on CollapseAll/ExpandAll buttons click `Uncaught TypeError: Cannot read properties of null (reading 'style')` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-15 12:54:39 +02:00
Aliaksandr Valialkin	e2be41f4dd	docs/CHANGELOG.md: document `ad6bdd78d0`	2022-02-15 12:48:23 +02:00
Corporte Gadfly	cf8df24227	match fileSDCheckInterval with prometheus file_sd_config default (#2188 )	2022-02-15 12:05:57 +02:00
Roman Khavronenko	3458a3d593	Monitoring cluster (#2191 ) * dashboards: add `CPU percentage` panel for cluster dashboards The new panel `CPU percentage` was added instead if adding a limit to the existing `CPU` panel because dasbhoard may display big number of components each with own limits. The separate panel should provide a clear display of CPU load. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards: sync vmagent and vmalert changes from single version Signed-off-by: hagen1778 <roman@victoriametrics.com> * docker: remove unsupported param from vmagent config Signed-off-by: hagen1778 <roman@victoriametrics.com> * alerts: add `TooHighCPUUsage` alert for all VM components Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-15 11:57:58 +02:00
Aliaksandr Valialkin	38c73a00db	docs/CHANGELOG.md: cut v1.73.0	2022-02-14 17:54:53 +02:00
Aliaksandr Valialkin	5d8ea8c918	docs/CHANGELOG.md: document `3d890e89f1`	2022-02-14 17:42:33 +02:00
Nikolay	748034e7af	Adds server certificate reload for lib/http (#2186 ) * Adds server certificate reload for lib/http https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171 * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-14 17:42:33 +02:00
Nikolay	48a9e068be	adds release build for macos darwin amd64 and arm64 (#2185 ) * adds release build for macos darwin amd64 and arm64 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1896 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1851 * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-14 17:42:33 +02:00
Aliaksandr Valialkin	c723ac6a8f	app/vmui: small fixes * Remove unneeded dependency on `numeral` package * Properly parse numbers obtained from /api/v1/query_range according to https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats * Optimize updating processing the received data from /api/v1/query_range * Make smoother zoom on `ctrl+scroll` * Reduce the number of points received from /api/v1/query_range by 2x in order to reduce load on backend	2022-02-14 16:31:31 +02:00
Aliaksandr Valialkin	fc57035343	deployment/docker/docker-compose.yml: update Grafana from v8.3.4 to v8.3.5 See https://grafana.com/blog/2022/02/08/grafana-7.5.15-and-8.3.5-released-with-moderate-severity-security-fixes	2022-02-14 13:22:06 +02:00
Aliaksandr Valialkin	fc0771a888	docs/CHANGELOG.md: document `c90c1c4d54`	2022-02-14 13:13:03 +02:00
Nikolay	c11d0949c8	fixes all_tenants query option usage for openstack service discovery (#2184 ) explicit use configuration parametr instead of conditional add https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182	2022-02-14 13:13:03 +02:00
dependabot[bot]	a2bbd08229	build(deps): bump @types/node in /app/vmui/packages/vmui (#2183 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.16 to 17.0.17. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-14 13:13:03 +02:00
dependabot[bot]	beedce6eac	build(deps): bump follow-redirects in /app/vmui/packages/vmui (#2180 ) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.6 to 1.14.8. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.6...v1.14.8) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-14 13:13:03 +02:00
Aliaksandr Valialkin	31b42e9c57	lib/promscrape: add `expand all` and `collapse all` buttons to `/targets` page See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-12 18:42:01 +02:00
Aliaksandr Valialkin	53c2135d2a	lib/storage: tune the logic for pre-populating of the per-day inverted index for the next day - Postpone the pre-poulation to the last hour of the current day. This should reduce the number of useless entries in the next per-day index, which shouldn't be created there, when the corresponding time series are stopped to be pushed during the current day. - Make the pre-population more smooth in time by using the hash of MetricID instead of MetricID itself when calculating the need for for the given MetricID pre-population. - Sync the logic for pre-population of the next day inverted index with the logic of pre-populating tsid cache after indexdb rotation. This should improve code maintainability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401	2022-02-12 16:39:33 +02:00
Aliaksandr Valialkin	989668beba	app/vmselect/promql: return at most one time series from `absent_over_time()` in the same way as Prometheus does See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130	2022-02-12 15:46:43 +02:00
Aliaksandr Valialkin	70cfb87c0e	deployment/docker: update Go builder from v1.17.6 to v1.17.7 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.7+label%3ACherryPickApproved	2022-02-12 01:14:07 +02:00
Aliaksandr Valialkin	916989479e	vendor: `make vendor-update`	2022-02-12 01:14:06 +02:00
Aliaksandr Valialkin	216b13186d	docs/vmagent.md: add `unsupported Prometheus config sections` chapter	2022-02-12 01:08:34 +02:00
Aliaksandr Valialkin	3ab4aef140	docs/CHANGELOG.md: document `ea153e5f90`	2022-02-12 00:48:40 +02:00
artifactori	943fc056ca	Show gce sdconfig zone on vmagent:8429/config (#2178 ) * vmagent: add test for marshalling gce sdconfig with ZoneYAML * vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig	2022-02-12 00:48:38 +02:00
Roman Khavronenko	d107f86fbc	lib/index: reduce read/write load after indexDB rotation (#2177 ) * lib/index: reduce read/write load after indexDB rotation IndexDB in VM is responsible for storing TSID - ID's used for identifying time series. The index is stored on disk and used by both ingestion and read path. IndexDB is stored separately to data parts and is global for all stored data. It can't be deleted partially as VM deletes data parts. Instead, indexDB is rotated once in `retention` interval. The rotation procedure means that `current` indexDB becomes `previous`, and new freshly created indexDB struct becomes `current`. So in any time, VM holds indexDB for current and previous retention periods. When time series is ingested or queried, VM checks if its TSID is present in `current` indexDB. If it is missing, it checks the `previous` indexDB. If TSID was found, it gets copied to the `current` indexDB. In this way `current` indexDB stores only series which were active during the retention period. To improve indexDB lookups, VM uses a cache layer called `tsidCache`. Both write and read path consult `tsidCache` and on miss the relad lookup happens. When rotation happens, VM resets the `tsidCache`. This is needed for ingestion path to trigger `current` indexDB re-population. Since index re-population requires additional resources, every index rotation event may cause some extra load on CPU and disk. While it may be unnoticeable for most of the cases, for systems with very high number of unique series each rotation may lead to performance degradation for some period of time. This PR makes an attempt to smooth out resource usage after the rotation. The changes are following: 1. `tsidCache` is no longer reset after the rotation; 2. Instead, each entry in `tsidCache` gains a notion of indexDB to which they belong; 3. On ingestion path after the rotation we check if requested TSID was found in `tsidCache`. Then we have 3 branches: 3.1 Fast path. It was found, and belongs to the `current` indexDB. Return TSID. 3.2 Slow path. It wasn't found, so we generate it from scratch, add to `current` indexDB, add it to `tsidCache`. 3.3 Smooth path. It was found but does not belong to the `current` indexDB. In this case, we add it to the `current` indexDB with some probability. The probability is based on time passed since the last rotation with some threshold. The more time has passed since rotation the higher is chance to re-populate `current` indexDB. The default re-population interval in this PR is set to `1h`, during which entries from `previous` index supposed to slowly re-populate `current` index. The new metric `vm_timeseries_repopulated_total` was added to identify how many TSIDs were moved from `previous` indexDB to the `current` indexDB. This metric supposed to grow only during the first `1h` after the last rotation. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-12 00:34:44 +02:00
Aliaksandr Valialkin	21a42e990f	lib/storage: fix broken BenchmarkHeadPostingForMatchers for `{i=~".*"}` after `f4dead529f` The commit `f4dead529f` makes such query to return nothing instead of all the time series. This aligns more with Prometheus behaviour.	2022-02-12 00:28:21 +02:00
Roman Khavronenko	791cad8c2e	lib/promscrape: support prometheus-like duration in scrape configs (#2169 ) * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:51 +02:00
Aliaksandr Valialkin	727b29d4a3	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_endpointslice_{label,annotation}*` labels to be consistent with other `role` values for Kubernetes service discovery	2022-02-11 14:56:10 +02:00
Nikolay	265938a385	fixes service discovery for kubernetes (#2173 ) * fixes service discovery for kubernetes now it must take in account all pods that belong to the discovered endpoint and endpointslice adds simple test for endpoints https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134 * wip * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin	895c9f4f11	lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin	102c9a4bf9	lib/blockcache: use higher number of shards for higher number of CPU cores This should reduce mutex contention and increase performance Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:11 +02:00
Aliaksandr Valialkin	6cb2954612	docs/CHANGELOG.md: document `4e722c459b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2167	2022-02-10 12:21:05 +02:00
Aliaksandr Valialkin	718342a806	docs/Articles.md: add an article from Asserts.io, which mentions VictoriaMetrics components	2022-02-10 12:21:05 +02:00
hagen1778	1662b5bcec	vmalert: fix bug with relative links in UI https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2167 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-10 12:21:05 +02:00
Aliaksandr Valialkin	ad19c9d302	lib/promscrape: fix errors in test config The errors were discovered after enabling strict parse mode by default. See `9bb60ab00f`	2022-02-08 20:10:28 +02:00
Aliaksandr Valialkin	fae3040868	lib/blockcache: split the cache into multiple shards This should reduce contention on cache mutex on hosts with many CPU cores, which, in turn, should increase overall throughput for the cache. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-08 19:48:32 +02:00
Aliaksandr Valialkin	a0a56d6c1c	lib/mergeset: tune sizes for `indexdb/dataBlocks` and `indexdb/indexBlocks` according to production workload This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742	2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin	d1c17fe385	app/vmselect/vmui: `make vmui-update`	2022-02-08 17:29:35 +02:00
dependabot[bot]	2da5acf6c2	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2166 ) Bumps [@mui/icons-material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-icons-material) from 5.3.1 to 5.4.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.1/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:24:14 +02:00
dependabot[bot]	069f0d2398	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#2165 ) Bumps [@mui/styles](https://github.com/mui/material-ui/tree/HEAD/packages/mui-styles) from 5.3.0 to 5.4.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.1/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:24:14 +02:00
dependabot[bot]	aa7005adc2	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2164 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.10.1 to 5.11.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.11.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:24:14 +02:00
dependabot[bot]	8de07b2784	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2162 ) Bumps [@mui/material](https://github.com/mui/material-ui/tree/HEAD/packages/mui-material) from 5.3.1 to 5.4.1. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/v5.4.1/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	b9cdea9020	build(deps-dev): bump @typescript-eslint/parser (#2159 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.10.1 to 5.11.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.11.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	fece4074de	build(deps): bump uplot from 1.6.18 to 1.6.19 in /app/vmui/packages/vmui (#2163 ) Bumps [uplot](https://github.com/leeoniya/uPlot) from 1.6.18 to 1.6.19. - [Release notes](https://github.com/leeoniya/uPlot/releases) - [Commits](https://github.com/leeoniya/uPlot/compare/1.6.18...1.6.19) --- updated-dependencies: - dependency-name: uplot dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	8efdb8e408	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2161 ) Bumps [@mui/lab](https://github.com/mui/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.66 to 5.0.0-alpha.68. - [Release notes](https://github.com/mui/material-ui/releases) - [Changelog](https://github.com/mui/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	f5438e8d9b	build(deps): bump @types/node in /app/vmui/packages/vmui (#2160 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.13 to 17.0.16. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	e069afb4f4	build(deps): bump @types/react in /app/vmui/packages/vmui (#2147 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.38 to 17.0.39. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
dependabot[bot]	0f00b8c711	build(deps): bump @testing-library/jest-dom in /app/vmui/packages/vmui (#2146 ) Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.16.1 to 5.16.2. - [Release notes](https://github.com/testing-library/jest-dom/releases) - [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/jest-dom/compare/v5.16.1...v5.16.2) --- updated-dependencies: - dependency-name: "@testing-library/jest-dom" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-02-08 17:21:37 +02:00
Aliaksandr Valialkin	b81e6ba403	docs/CHANGELOG.md: add instructions on how to build VictoriaMetrics components from source code in order to test tip changes	2022-02-08 16:44:20 +02:00
Nikolay	feefbbab48	adds CGO build for arm64 (#2102 ) * adds CGO build for arm64 it must improve performance for arm64 based deployments of vmstorage and vmsingle for 15-20% it depends on gozstd package update for correct musl gozstd vendoring * typo fixes * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-08 16:26:55 +02:00
Roman Khavronenko	763cb68a67	docs: add Storage section to docs (#2155 ) * docs: add Storage section to docs https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2151 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * docs: `make docs-sync` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-08 16:00:18 +02:00
Aliaksandr Valialkin	eed66b6640	lib/promscrape: set `-promscrape.config.strictParse` to true by default This allows detecting long-living silent errors in -promscrape.config	2022-02-08 15:42:33 +02:00
Aliaksandr Valialkin	8863339b6b	lib/blockcache: `make fmt`	2022-02-08 15:42:31 +02:00
Aliaksandr Valialkin	6e05e3308e	docs/Single-server-VictoriaMetrics.md: add a link to exlanation on how downsampling works	2022-02-08 15:11:27 +02:00
Aliaksandr Valialkin	1caee74235	lib/blockcache: eliminate possible race when Cache.Put is called for the same entry from multiple goroutines The race could result in incorrect cache size tracking, which, in turn, could result in too frequent cache cleaning	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	10476738a8	lib/blockcache: increase the lifetime for rarely accessed blocks from 2 minutes to 5 minutes This should improve data ingestion speed if time series samples are ingested with interval bigger than 2 minutes. The actual interval could exceed 2 minutes if the original interval between samples doesn't exceed 2 minutes in the case of slow inserts. Slow inserts may appear in the following cases: * Big number of new time series are pushed to VictoriaMetrics, so they couldn't be registered in 2 minutes. * MetricName->tsid cache reset on indexdb rotation or due to unclean shutdown. In this case VictoriaMetrics needs to load MetricName->tsid entries for all the incoming series from IndexDB. IndexDB uses the block cache for increasing lookup performance. If the cache has no the needed block, then IndexDB reads and unpacks the block from disk. This requires an extra disk read IO and CPU. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This also should increase performance for periodically executed queries with intervals from 2 minutes to 5 minutes. See the previous similar commit - `43103be011` It is possible that the timeout can be increased further. Let's collect production numbers for this change so the timeout could be adjusted further.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	b7cefff7b0	lib/workingsetcache: use the original cache size limits when rotating caches Previously limits for new caches were taken from cache stats. These limits could mismatch the original limits. This could result in failed cache load if the stored cache has been created with the limits obtained from cache stats.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	178178bdd4	vendor: update github.com/VictoriaMetrics/fastcache from v1.8.0 to v1.9.0	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	87071640a7	lib/blockcache: return proper number of entries from the cache This has been broken in `0d7374ad2f`	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	75c5c33694	vendor: update github.com/valyala/gozstd from v1.15.1 to v1.16.0	2022-02-07 18:31:13 +02:00
Aliaksandr Valialkin	2149805886	docs/CHANGELOG.md: add links to issues, which could benefit from improved re-routing algorithm	2022-02-07 16:54:27 +02:00
Aliaksandr Valialkin	eb466e0c4e	docs: fix links to FAQ.md entries after they have been changed in `ccf04239e6`	2022-02-07 16:01:34 +02:00
Aliaksandr Valialkin	755e26e67b	app/vminsert/netstorage: throttle warning logs, which can be too verbose when vminsert cannot send data to vmstorage nodes	2022-02-07 15:39:06 +02:00
Aliaksandr Valialkin	b4b40774ec	vendor: `make vendor-update`	2022-02-07 15:07:43 +02:00
Aliaksandr Valialkin	9fb5ce5fb6	app/vminsert/netstorage: log vmstorage addr, which cannot accept new samples due to overload and/or unavailability	2022-02-07 15:03:05 +02:00
Aliaksandr Valialkin	5da7f08be6	docs: cross-link downsampling docs from deduplication and vmalert docs	2022-02-07 14:53:50 +02:00
Aliaksandr Valialkin	34d14c4940	all: substitute zeroTime with time.Time{}, since this generates more optimal binary code	2022-02-07 14:36:41 +02:00
Aliaksandr Valialkin	5aee6eb406	app/vminsert/netstorage: tune re-routing algorithm further	2022-02-07 14:35:39 +02:00
Aliaksandr Valialkin	1a5546006d	app/vminsert/netstorage: typo fix after `021ee53ba8`	2022-02-07 13:20:43 +02:00
Aliaksandr Valialkin	e2d12a25e0	lib/netutil: increase dial timeout from 1 second to 5 seconds There are real-world cases when TCP connection needs more than 1 second to be established.	2022-02-07 12:33:40 +02:00
Aliaksandr Valialkin	c738739494	app/vminsert: add `-dropSamplesOnOverload` command-line flag Drop incoming samples if the destination vmstorage node is unavailable and/or accepts data at slower rate than other vmstorage nodes	2022-02-07 12:32:18 +02:00
Aliaksandr Valialkin	021ee53ba8	app/vminsert: improve re-routing logic in order to spread rows more evenly among the available storage nodes	2022-02-06 20:20:02 +02:00
Aliaksandr Valialkin	d24e5d9efd	lib/promscrape: show the total number of scrapes and the total number of scrape errors per target at /targets page This information may be useful when debugging unreliable scrape targets	2022-02-03 20:23:27 +02:00
Aliaksandr Valialkin	678b3e71db	lib/promscrape: provide the ability to fetch target responses on behalf of vmagent or single-node VictoriaMetrics This feature may be useful when debugging metrics for the given target located in isolated environment	2022-02-03 19:02:12 +02:00
Aliaksandr Valialkin	1173964d8d	app/vmselect/promql: clarify comments on why the right side of `if` and `and` operators are executed at first	2022-02-03 00:26:42 +02:00
Aliaksandr Valialkin	10367d7e4c	app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values Such filters may slow down time series search, so just skip them. This is a follow-up for `e7f1ceeb84` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827	2022-02-02 23:42:25 +02:00
Aliaksandr Valialkin	97b7b94f91	docs/CHANGELOG.md: document `55e3bbd4cc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1567	2022-02-02 23:42:25 +02:00
Aliaksandr Valialkin	dbead4813e	docs: updates after `5da71eb685` * Mention about the ability to configure vmalert notifiers via files in docs/CHANGELOG.md * Mention about the ability to use Consul service discovery for vmalert notifiers in docs/CHANGELOG.md * Run `make docs-sync` in order to sync app/vmalert/README.md to docs/vmalert.md	2022-02-02 23:42:25 +02:00
Pablo Ley	66271bfeee	Added missing links in EmbeddedHPA API docs	2022-02-02 23:42:25 +02:00
hagen1778	a67e843257	vmalert: add support of `-notifier.basicAuth.passwordFile` flag for notifiers https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1567 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-02 23:42:25 +02:00
hagen1778	605de7bdd6	vmalert: remove trailing slash for static notifier addresses This would make addresses `http://localhost:9093` and `http://localhost:9093/` both to result into `http://localhost:9093/api/v2/alerts`. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-02-02 23:42:25 +02:00
Roman Khavronenko	2a3a62dc41	vmalert: support configuration file for notifiers (#2127 ) vmalert: support configuration file for notifiers * vmalert notifiers now can be configured via file see https://docs.victoriametrics.com/vmalert.html#notifier-configuration-file * add support of Consul service discovery for notifiers config see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1947 * add UI section for currently loaded/discovered notifiers * deprecate `-rule.configCheckInterval` in favour of `-configCheckInterval` * add ability to suppress logs for duplicated targets for notifiers discovery * change behaviour of `vmalert_alerts_send_errors_total` - it now accounts for failed alerts, not HTTP calls.	2022-02-02 23:42:25 +02:00
Aliaksandr Valialkin	5f266370c5	all: follow-up after `4bdd10ab90` Properly use new bytesutil.Resize* functions	2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin	13c2692ca1	app/vmselect/promql: properly handle `foo or bar` queries Such queries may miss `bar` results after the commit `e7f1ceeb84` because common label filters from `foo` could be mistakenly applied to `bar`.	2022-02-01 17:43:10 +02:00
Aliaksandr Valialkin	2cc5f2940f	vendor: update github.com/VictoriaMetrics/metricsql from v0.39.0 to v0.40.0	2022-02-01 17:43:09 +02:00
Aliaksandr Valialkin	d8d59ff760	lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	02b2bfcff3	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations Follow-up for `f4989edd96` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	084664d780	lib/encoding: substitute `64-bits.LeadingZeros64()` with `bits.Len64()`	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	0fbfa8c245	lib/storage: avoid allocations of tsidPrev on every blockStreamReader.NextBlock() call This is a follow-up for `00b7c97d2a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-31 22:47:16 +02:00
Aliaksandr Valialkin	784cdcb080	app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first PromQL and MetricsQL operators are case-insensitive	2022-01-31 22:07:35 +02:00
Aliaksandr Valialkin	a02dde6cc7	lib/cgroup: fall back to runtime.NumCPU() when determining process_cpu_cores_available metric if it is impossible to determine cpu quota via cgroups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:31:12 +02:00
Aliaksandr Valialkin	566e12874d	lib/cgroup: expose `process_cpu_cores_available` metric This metric shows the number of CPU cores available to the process. This allows creating alerting rules on CPU saturation with the following query: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	04d6596298	app/vmselect/promql: optimize queries, which join on `_info` metrics. Automatically add common filters from one side of binary operation to the other side before sending the query to storage subsystem. See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/ and https://www.robustperception.io/exposing-the-software-version-to-prometheus	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	ad5d646e4e	docs/Release-Guide.md: add instructions on how to create signed release tags	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	f3250307b1	vendor: `make vendor-update`	2022-01-31 12:46:44 +02:00
Aliaksandr Valialkin	35164d4dcf	docs/CHANGELOG.md: document `6a519896db`	2022-01-31 12:42:27 +02:00
Aliaksandr Valialkin	d95d945046	app/vmselect/vmui: `make vmui-update`	2022-01-31 12:42:21 +02:00
Yury Molodov	f6a76e7bfd	fix: change logic set free fields for legend (#2117 )	2022-01-31 12:42:21 +02:00
dependabot[bot]	76421f5535	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2129 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.3.0 to 5.3.1. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.3.1/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	80068504db	build(deps): bump preact in /app/vmui/packages/vmui (#2128 ) Bumps [preact](https://github.com/preactjs/preact) from 10.6.4 to 10.6.5. - [Release notes](https://github.com/preactjs/preact/releases) - [Commits](https://github.com/preactjs/preact/compare/10.6.4...10.6.5) --- updated-dependencies: - dependency-name: preact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	25db81459c	build(deps-dev): bump @typescript-eslint/parser (#2123 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.10.0 to 5.10.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.10.1/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	ab526bf18c	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2122 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.3.0 to 5.3.1. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.3.1/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	7d3b0b7c44	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2124 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.10.0 to 5.10.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.10.1/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	8d2f113a8a	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2125 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.65 to 5.0.0-alpha.66. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
dependabot[bot]	51fe0c3336	build(deps): bump @types/node in /app/vmui/packages/vmui (#2126 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.10 to 17.0.13. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-31 12:42:21 +02:00
Aliaksandr Valialkin	776b7bc9f8	lib/storage/table.go: add missing `tb.ptwsLock.Unlock()` before the return This is a follow-up for `a1083d0531` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2103	2022-01-28 12:12:58 +02:00
匠心零度	a1083d0531	optimized code (#2103 ) * optimized code ,because only the first error,so no need var errors []error * optimized code ,because only the first error,so no need var errors []error Co-authored-by: lirenzuo <lirenzuo@shein.com>	2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin	069880fd3c	docs/vmauth.md: mention that backend services must be accessible only via vmauth	2022-01-28 12:07:12 +02:00
Denys Holius	9e7d6a585b	FAQ update (#2111 ) * FAQ update: how downsampling and deduplication will work at the same time * Update docs/FAQ.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-01-27 20:36:15 +02:00
jaysony	3e22537248	correct VMAgentStatus description (#2109 ) correct VMAgentStatus description from VMAlert to VMAgent.	2022-01-27 20:18:37 +02:00
Aliaksandr Valialkin	0ac2a51682	vendor: update github.com/VictoriaMetrics/metricsql from v0.37.0 to v0.38.0 This adds more optimization cases for https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization For example: * Multi-level transform functions. For example, abs(round(foo{a="b"})) + bar{x="y"} is now optimized to abs(round(foo{a="b",x="y"})) + bar{a="b",x="y"} * Binary operations with `on()`, `without()`, `group_left()` and `group_right()` modifiers. For example, foo{a="b"} on (a) + bar is now optimized to foo{a="b"} on (a) + bar{a="b"} * Multi-level binary operations. For example, foo{a="b"} + bar{x="y"} + baz{z="q"} is now optimized to foo{a="b",x="y",z="q"} + bar{a="b",x="y",z="q"} + baz{a="b",x="y",z="q"} * Aggregate functions. For example, sum(foo{a="b"}) by (c) + bar{c="d"} is now optimized to sum(foo{a="b",c="d"}) by (c) + bar{c="d"}	2022-01-27 19:04:45 +02:00
Aliaksandr Valialkin	dcf20c7aa1	vendor: `make vendor-update`	2022-01-27 13:22:23 +02:00
Denys Holius	588f0ab4ee	Update version of golangci-lint Update version of golangci-lint from v1.43.0 to v1.44.0 in Makefile. See https://github.com/golangci/golangci-lint/releases/tag/v1.44.0 .	2022-01-27 13:22:22 +02:00
Aliaksandr Valialkin	a3eafd2e7f	app/vmselect/vmui: `make vmui-update` after `e3995572bb`	2022-01-25 17:52:37 +02:00
Yury Molodov	0b1db9fafe	feat: add sortable columns (#2101 )	2022-01-25 17:52:35 +02:00
Aliaksandr Valialkin	4911c0c045	docs/operator: fix links to Operator API docs	2022-01-25 17:38:41 +02:00
Aliaksandr Valialkin	6232eaa938	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin	7ec0705b98	lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock This should reduce the memory required for indexdb/dataBlocks cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin	b5abea0e36	app/vmselect/vmui: `make vmui-update`	2022-01-24 12:25:00 +02:00
dependabot[bot]	6f44fe34f6	build(deps-dev): bump @typescript-eslint/parser (#2097 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.9.1 to 5.10.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.10.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	bf3d4425d0	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2096 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.64 to 5.0.0-alpha.65. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	870261e650	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#2095 ) Bumps [@mui/styles](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-styles) from 5.2.3 to 5.3.0. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.3.0/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	7eb1d587ed	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2094 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.8 to 5.3.0. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.3.0/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	286b3178f1	build(deps): bump web-vitals in /app/vmui/packages/vmui (#2093 ) Bumps [web-vitals](https://github.com/GoogleChrome/web-vitals) from 2.1.3 to 2.1.4. - [Release notes](https://github.com/GoogleChrome/web-vitals/releases) - [Changelog](https://github.com/GoogleChrome/web-vitals/blob/main/CHANGELOG.md) - [Commits](https://github.com/GoogleChrome/web-vitals/compare/v2.1.3...v2.1.4) --- updated-dependencies: - dependency-name: web-vitals dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	9c5ae51661	build(deps): bump @types/node in /app/vmui/packages/vmui (#2092 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.8 to 17.0.10. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	808b617067	build(deps): bump typescript in /app/vmui/packages/vmui (#2091 ) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.5.4 to 4.5.5. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](https://github.com/Microsoft/TypeScript/compare/v4.5.4...v4.5.5) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	2e703de99e	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2090 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.9.1 to 5.10.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.10.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
dependabot[bot]	37aff7ccc7	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2089 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.2.5 to 5.3.0. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.3.0/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-24 12:17:48 +02:00
Aliaksandr Valialkin	65176726b3	lib/logger: removed broken test after `746ee191e8`	2022-01-24 12:15:11 +02:00
Aliaksandr Valialkin	49650fe6aa	lib/logger/throttler.go: show the original location of the error and warning message Previously the location inside LogThrottler implementation was shown. This could complicate debugging.	2022-01-23 13:55:48 +02:00
Aliaksandr Valialkin	53440226d6	vendor: `make vendor-update`	2022-01-23 13:15:37 +02:00
f41gh7	2f56f484af	updates operator docs	2022-01-23 13:15:37 +02:00
Aliaksandr Valialkin	233101137d	lib/blockcache: optimize blockcache a bit - Optimize Cache.RemoveBlocksFromPart(), so it doesn't need to iterate over all the cached blocks. - Cache blocks if there were no cache misses during the last 2 minutes. This may be the case when new blocks are added simultaneously to the storage and to the cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-23 13:08:55 +02:00
Yury Molodov	196bef8348	vmui: fixed display type switching (#2088 ) * fix: correct switch display type * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 16:57:14 +02:00
Yury Molodov	be8a55477a	feat: copy pairs on click in legend (#2087 )	2022-01-21 16:57:12 +02:00
Aliaksandr Valialkin	9edf407144	lib/mergeset: tune caches size limits for `indexdb/dataBlocks` and `indexdb/indexBlocks` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin	4e05298756	lib/storage: properly limit cardinality when ingesting multiple samples for the same time series in a single request	2022-01-21 12:38:22 +02:00
Yurii Kravets	4b7650832e	Created Operator folder in docs (#2034 ) * Created Operator folder in docs Transferred Operator documentation * Removed Contributing and Release * Changed sort numbering * Renamed folder Operator -> operator * 1 1 * Name change Operator -> operator * Removed colon symbol * Useful links transformed to links style * "updated at..." is no longer a header * delete manager patch.yaml * delete kustomization.yaml * removed part with links * community and contributions part removed * Delete readme * Docs navigation removed	2022-01-21 12:06:41 +02:00
Yury Molodov	034012c80f	vmui: fix time range selector (#2085 ) * fix: add date validate for time range * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-21 12:03:41 +02:00
Aliaksandr Valialkin	93cadb4a16	all: update link to EULA from https://victoriametrics.com/assets/VM_EULA.pdf to https://victoriametrics.com/legal/eula/	2022-01-20 22:22:44 +02:00
Aliaksandr Valialkin	e3277918e4	lib/storage: verify that blocks in a single part are sorted by TSID when reading sequential blocks from the part This may help narrowing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:28 +02:00
Aliaksandr Valialkin	54ee71e16d	lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used. This may help nailing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:24 +02:00
Aliaksandr Valialkin	5159a9451f	lib/blockcache: add missing dependency after `145337792d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:51:03 +02:00
Aliaksandr Valialkin	6ae584b9b3	lib/{mergeset,storage}: properly limit cache sizes for indexdb Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`, since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded, which could result to out of memory errors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin	c2abd6a702	docs/CHANGELOG.md: document the bugfix for highestMax() function is Graphite render API	2022-01-20 12:16:12 +02:00
Aliaksandr Valialkin	d9ab623c5b	docs/Release-Guide.md: update instructions for exporting snapcraft login to ~/.snap/login.json	2022-01-19 15:21:49 +02:00
Aliaksandr Valialkin	6233e7c40c	docs/CHANGELOG.md: add missing parens in example for `@` modifier	2022-01-19 13:05:29 +02:00
Oct	9cd8aa8444	docs: Correct some urls in the docs (#2081 ) * docs:update Swift S3 api url * docs/vmbackup.md: update Swift S3 api url * docs: update KairosDB docs url * docs: update thanos docs url	2022-01-19 11:57:02 +02:00
Aliaksandr Valialkin	2eaf7a7c46	docs/CHANGELOG.md: fix incorrect link to the issue Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911	2022-01-19 00:08:31 +02:00
Aliaksandr Valialkin	e8900c90a8	app/vmui: fix `last 6 months` time range picker Previously it was incorrectly selecting 6 minutes instead of 6 months. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960	2022-01-18 23:28:49 +02:00
Aliaksandr Valialkin	f3196e48e1	docs/CHANGELOG.md: cut v1.72.0	2022-01-18 22:43:34 +02:00
Aliaksandr Valialkin	8dde20091f	deployment/docker: update Grafana from v8.3.2 to v8.3.4	2022-01-18 22:42:32 +02:00
Aliaksandr Valialkin	0c9cf79e51	docs/vmbackup.md: `make docs-sync` after `ca11def2a5`	2022-01-18 22:30:23 +02:00
Aliaksandr Valialkin	276665d0ae	app/vmgateway/README.md: sync with enterprise vesion after adding `extra_filters` handling Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2022-01-18 22:30:23 +02:00
Roman Khavronenko	c71a05a528	docs: update retention docs with additional details (#2060 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-01-18 22:30:23 +02:00
Aliaksandr Valialkin	f4a9ae28f3	app/vmui: properly calculate graph range for y axis Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037	2022-01-18 22:23:40 +02:00
Aliaksandr Valialkin	f31776d0e4	app/vmui: reduce the refresh interval during graph scrolling/zooming from 1 second to 300 milliseconds One second feels too laggy, so let's reduce the refresh interval to 300 milliseconds. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064	2022-01-18 22:23:40 +02:00
Yury Molodov	1d19303f35	fix: remove buffer period (#2078 ) * fix: remove buffer period * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the implemented feature Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 22:23:40 +02:00
Yury Molodov	5b286a4728	vmui: correct url encoding (#2067 ) * fix: correct encode multi-line queries * fix: change autocomplete for correct arrows work * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document the bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:32:30 +02:00
Oct	cb5d8a8633	docs: Correct config URL (#2077 ) docs: update incorrect links	2022-01-18 21:27:26 +02:00
Aliaksandr Valialkin	ce55b31d2e	docs/CHANGELOG.md: document `8e3f9c1fbb`	2022-01-18 21:27:26 +02:00
Yury Molodov	d4ccd48faa	vmui: correct calc axes limits (#2058 ) * fix: correct calc axes limits * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 21:27:26 +02:00
Aliaksandr Valialkin	7dabfbee45	docs/CHANGELOG.md: document `fcd33fc409`	2022-01-18 12:47:02 +02:00
Yury Molodov	e548fc9021	vmui: change layout (#2054 ) * fix: change query reset * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * feat: move time selector to Header * checkout * fix: remove unused vars * update package-lock.json * fix: correct styles * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-18 12:47:01 +02:00
Aziz Köksal	f3f764fe8b	docs: proofread FAQ.md (#2062 ) * Added missing articles. * Changed minus characters to long dashes. * Rephrased some words. * Corrected grammar mistakes. * Set correct possessive apostrophes. * "setup" -> "set up" * "no need in" -> "no need for" * "comparing" -> "compared" * "less" -> "fewer" * "additionally" -> "in addition to"	2022-01-18 12:40:43 +02:00
Aliaksandr Valialkin	b020bf4e3f	docs/CHANGELOG.md: document that vmgateway now supports extra_filters option Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2022-01-18 12:38:24 +02:00
Aliaksandr Valialkin	4f3ed98ea4	docs/vmgateway.md: update docs, since vmgateway now supports extra_filters See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2022-01-18 12:38:23 +02:00
Aliaksandr Valialkin	e8706ce417	vendor: `make vendor-update`	2022-01-17 15:45:47 +02:00
Aliaksandr Valialkin	cc76174387	app/vmselect/promql: properly keep metric names when optimized path is used for aggregate function calculations For example, `sum(rate(...) keep_metric_names) by (__name__)` didn't leave the original metric name because of this issue. This is a follup-up for `1bdc71d917` Udates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-17 15:33:39 +02:00
dependabot[bot]	34da060cd1	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2075 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.7 to 5.2.8. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.8/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-17 15:33:39 +02:00
dependabot[bot]	6c6be1cb84	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2076 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.9.0 to 5.9.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.9.1/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-17 15:33:39 +02:00
dependabot[bot]	962a2e0049	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2074 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.63 to 5.0.0-alpha.64. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.old.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-17 15:33:39 +02:00
dependabot[bot]	6a9d7e291a	build(deps-dev): bump @typescript-eslint/parser (#2073 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.9.0 to 5.9.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.9.1/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-17 15:33:39 +02:00
dependabot[bot]	4d04cd7a9b	build(deps): bump qs from 6.10.2 to 6.10.3 in /app/vmui/packages/vmui (#2072 ) Bumps [qs](https://github.com/ljharb/qs) from 6.10.2 to 6.10.3. - [Release notes](https://github.com/ljharb/qs/releases) - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](https://github.com/ljharb/qs/compare/v6.10.2...v6.10.3) --- updated-dependencies: - dependency-name: qs dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-17 15:33:39 +02:00
Aliaksandr Valialkin	5a1e4a140f	app/vmselect/promql: implement `keep_metric_names` modifier for transform and rollup functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949	2022-01-14 04:15:35 +02:00
Aliaksandr Valialkin	53c9a81f2b	app/vmselect/promql: add `stale_samples_over_time()` function	2022-01-14 01:49:24 +02:00
Aliaksandr Valialkin	410905e2ed	docs/CHANGELOG.md: yet another attempt to fix formatting for yaml snippet	2022-01-14 01:15:15 +02:00
Aliaksandr Valialkin	2eb9ec4476	docs/CHANGELOG.md: fix formatting for `scrape_configs` example	2022-01-14 01:03:02 +02:00
Aliaksandr Valialkin	da95516a1f	lib/promscrape: expose promscrape_stale_samples_created_total metric for monitoring the number of created stale samples	2022-01-14 01:00:40 +02:00
Aliaksandr Valialkin	653055b924	vendor: update github.com/valyala/gozstd from v1.15.0 to v1.15.1	2022-01-13 23:44:18 +02:00
Aliaksandr Valialkin	755779adb7	app/vmalert: add `parseDuration` function in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/8817	2022-01-13 23:30:56 +02:00
Aliaksandr Valialkin	dd91759f1f	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_node_provider_id` label for discovered Kubernetes nodes in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/9603	2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin	0f81ecd7df	app/vmalert: add `stripPort` template function in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/10002	2022-01-13 22:54:46 +02:00
Aliaksandr Valialkin	bc18368c15	lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881	2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin	0580a58feb	app/vmselect/promql: add support for `@` modifier Add support for `@` modifier in MetricsQL according to https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier Extend the support with the following features: * Allow using `@` modifier everywhere in the query. For example, `sum(foo) @ end()` * Allow using arbitrary expression as `@` modifier. For example, `foo @ (end() - 1h)` returns `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348	2022-01-13 22:15:19 +02:00
Aliaksandr Valialkin	7d635a2311	app/vmselect/promql: fix limit_offset() test The test has been broken in `addae7fc6a`	2022-01-13 16:58:47 +02:00
Aliaksandr Valialkin	972587e6f7	vendor: update github.com/valyala/gozstd from v1.14.2 to v1.15.0	2022-01-12 13:25:50 +02:00
Yury Molodov	7ee09286ef	vmui: switching to Preact (#2053 ) * feat: replace @codemirror to text field * feat: switch to Preact from React * fix: optimize mui imports * fix: remove unused vars * update package-lock.json * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-11 10:34:01 +02:00
Aliaksandr Valialkin	ad0c21ff26	vendor: `make vendor-update`	2022-01-11 10:16:01 +02:00
Aliaksandr Valialkin	de8299f465	lib/promscrape/discovery/dockerswarm: follow up after `68a117a25a` - Document the bugfix at docs/CHANGELOG.md - Set __address__ field after copying commonLabels to the resulting map of discovered labels. This makes sure that the correct __address__ label is used.	2022-01-11 09:22:03 +02:00
Alexander Shtuchkin	45a92e6ce1	Fix for #2038 : Make correct __address__ value for dockerswarm promscrape (#2041 )	2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin	21e5e90d75	docs/CHANGELOG.md: document `77bfa8181d`	2022-01-11 08:56:02 +02:00
Andrey Afoninsky	2e8c23e0f6	chore: add vmalert_remotewrite_total metric (#2040 ) Co-authored-by: Andrey Afoninsky <andrey.afoninsky@booking.com>	2022-01-11 08:56:02 +02:00
Dmitry Tolstoy	56fa9ede65	docs: Correct config URL (#2051 ) Missed /guides folder	2022-01-11 08:55:57 +02:00
dependabot[bot]	602f55886e	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2048 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.8.1 to 5.9.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.9.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	e521837ef0	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2044 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.62 to 5.0.0-alpha.63. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	0015622af0	build(deps): bump @types/node in /app/vmui/packages/vmui (#2049 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.6 to 17.0.8. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	ddc254e849	build(deps-dev): bump @typescript-eslint/parser (#2043 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.8.0 to 5.9.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.9.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	9d8445bb9e	build(deps): bump web-vitals in /app/vmui/packages/vmui (#2045 ) Bumps [web-vitals](https://github.com/GoogleChrome/web-vitals) from 2.1.2 to 2.1.3. - [Release notes](https://github.com/GoogleChrome/web-vitals/releases) - [Changelog](https://github.com/GoogleChrome/web-vitals/blob/main/CHANGELOG.md) - [Commits](https://github.com/GoogleChrome/web-vitals/compare/v2.1.2...v2.1.3) --- updated-dependencies: - dependency-name: web-vitals dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	71fc9ff4be	build(deps): bump @mui/material in /app/vmui/packages/vmui (#2046 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.4 to 5.2.7. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.7/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
dependabot[bot]	da98bda233	build(deps): bump @types/jest in /app/vmui/packages/vmui (#2047 ) Bumps [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest) from 27.0.3 to 27.4.0. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest) --- updated-dependencies: - dependency-name: "@types/jest" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-11 08:55:57 +02:00
Aliaksandr Valialkin	bc3718f719	deployment/docker: update Go builder from v1.17.5 to v1.17.6 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.6+label%3ACherryPickApproved	2022-01-07 13:34:15 +02:00
Aliaksandr Valialkin	f10b54a665	vendor: `make vendor-update`	2022-01-07 12:42:14 +02:00
Aliaksandr Valialkin	7509780cc0	docs/CHANGELOG.md: add release dates for every release	2022-01-07 12:28:01 +02:00
Aliaksandr Valialkin	fa89f3e5a5	lib/promscrape: do not send staleness markers on graceful shutdown This follows Prometheus behavior. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079	2022-01-07 01:19:06 +02:00
Aliaksandr Valialkin	32ef62b18f	docs/vmctl.md: `make docs-sync`	2022-01-07 01:19:04 +02:00
Aliaksandr Valialkin	c6abaa824d	docs/FAQ.md: add `what is the difference between vmagent and Prometheus agent` chapter	2022-01-06 23:14:22 +02:00
hagen1778	e405b29a83	vmctl: add option to rate limit data transfer speed The new flag `vm-rate-limit` defines data transfer speed limit in bytes per second. Rate limiting is not applied if flag is omitted. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1405 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-01-06 23:14:22 +02:00
Denis Golius	c74a6e8714	removed not needed directory	2022-01-06 23:14:22 +02:00
Aliaksandr Valialkin	13c36e5e54	docs: follow-up after `ae89b4e818`	2022-01-05 16:32:56 +02:00
Denys Holius	3a522c591a	Old links replaced for newest (#2033 ) * replaced old links to the website * fixed deletion main README.md file * fix: added docs files after docs-sync	2022-01-05 16:32:54 +02:00
Aliaksandr Valialkin	c91df022cc	docs/CHANGELOG.md: clarify the issue description, which is fixed by `38bf5fc136`	2022-01-05 16:19:32 +02:00
Aliaksandr Valialkin	80fc3fda07	lib/storage: follow-up for `38bf5fc136`	2022-01-05 16:02:17 +02:00
weng zhao	1e0fe615ad	vmstorage: fix query like `{foo=~"bar\|"}` return extra timeseries cause by negative filter transformation malfunction (#2032 ) 1. L2749 make kb.B remain the value of comonPrefix instead of tf.prefix 2. L2762 avoid change tf.value from "bar\|" to ".+r\|"	2022-01-05 15:57:54 +02:00
Aliaksandr Valialkin	c29ce8acef	docs/CHANGELOG.md: document `60266078ca`	2022-01-04 11:48:23 +02:00
Aliaksandr Valialkin	c461b39b21	Revert "Add check-rebased Github action (#2002 )" This reverts commit `2104330d4c`. This check doesn't work well for community pull requests, since third-party users aren't motivated to rebase pull requests to branch head after they are created. This check is useful for private repositories though.	2022-01-04 11:48:23 +02:00
John Seekins	edd2db1b10	Address some edge cases in OpenTSDB importer and speed it up (#2019 ) * Simplify queries to OpenTSDB (and make them properly appear in OpenTSDB query stats) and also tweak defaults a bit * Convert seconds to milliseconds before writing to VictoriaMetrics and increase subquery size Signed-off-by: John Seekins <jseekins@datto.com>	2022-01-04 11:48:23 +02:00
Aliaksandr Valialkin	276cccb888	docs/CHANGELOG.md: document `ac47733044`	2022-01-03 21:16:19 +02:00
Roman Khavronenko	e313e57090	vmctl: improve logging during import cancels/errors (#2006 ) On import process interruption `vmctl` now prints the max and min timestamps of: * last failed batch if import ended with error; * last sent batch if import was cancelled by user. To get more details for each timeseries in batch user needs to specify `--verbose` flag. The change does not relate to `vm-native` mode, since `vmctl` has no control over transferred data in this mode. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1236 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-01-03 21:16:18 +02:00
Aliaksandr Valialkin	cf5553d18e	app/vmselect/vmui: run `make vmui-update` after `89ff7b2465`	2022-01-03 21:06:34 +02:00
Yury Molodov	6ab46d2241	vmui: replace @codemirror to text field (#2003 ) * feat: replace @codemirror to text field * update package-lock.json Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-01-03 21:06:34 +02:00
dependabot[bot]	8dabd4136e	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2024 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.61 to 5.0.0-alpha.62. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 21:06:01 +02:00
dependabot[bot]	4e21269c78	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2028 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.8.0 to 5.8.1. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.8.1/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 21:05:57 +02:00
dependabot[bot]	fc499c03a2	build(deps): bump @types/node in /app/vmui/packages/vmui (#2027 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.1 to 17.0.6. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 21:05:57 +02:00
dependabot[bot]	6e281c076f	build(deps-dev): bump @babel/plugin-proposal-nullish-coalescing-operator (#2026 ) Bumps [@babel/plugin-proposal-nullish-coalescing-operator](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-proposal-nullish-coalescing-operator) from 7.16.5 to 7.16.7. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.16.7/packages/babel-plugin-proposal-nullish-coalescing-operator) --- updated-dependencies: - dependency-name: "@babel/plugin-proposal-nullish-coalescing-operator" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 21:05:57 +02:00
dependabot[bot]	536f5c710c	build(deps-dev): bump react-app-rewired in /app/vmui/packages/vmui (#2025 ) Bumps [react-app-rewired](https://github.com/timarney/react-app-rewired) from 2.1.9 to 2.1.11. - [Release notes](https://github.com/timarney/react-app-rewired/releases) - [Commits](https://github.com/timarney/react-app-rewired/compare/v2.1.9...v2.1.11) --- updated-dependencies: - dependency-name: react-app-rewired dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 21:05:57 +02:00
Yury Molodov	4f7ca2dd19	fix: change query reset (#2001 )	2022-01-03 20:56:12 +02:00
Aliaksandr Valialkin	22c78de2f2	docs/Quick-Start.md: simplify quick start guide	2022-01-03 17:29:10 +02:00
Aliaksandr Valialkin	fdfb92e134	LICENSE: update year from 2021 to 2022	2022-01-03 16:58:15 +02:00
dependabot[bot]	846a42316f	build(deps-dev): bump eslint-plugin-react in /app/vmui/packages/vmui (#2017 ) Bumps [eslint-plugin-react](https://github.com/yannickcr/eslint-plugin-react) from 7.27.1 to 7.28.0. - [Release notes](https://github.com/yannickcr/eslint-plugin-react/releases) - [Changelog](https://github.com/yannickcr/eslint-plugin-react/blob/master/CHANGELOG.md) - [Commits](https://github.com/yannickcr/eslint-plugin-react/compare/v7.27.1...v7.28.0) --- updated-dependencies: - dependency-name: eslint-plugin-react dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	d155e6506e	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#2014 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.2.4 to 5.2.5. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.5/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	ab0a3f0f71	build(deps-dev): bump @typescript-eslint/parser (#2015 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.7.0 to 5.8.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.8.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	219d4ee012	build(deps-dev): bump react-app-rewired in /app/vmui/packages/vmui (#2009 ) Bumps [react-app-rewired](https://github.com/timarney/react-app-rewired) from 2.1.8 to 2.1.9. - [Release notes](https://github.com/timarney/react-app-rewired/releases) - [Commits](https://github.com/timarney/react-app-rewired/compare/2.1.8...v2.1.9) --- updated-dependencies: - dependency-name: react-app-rewired dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	8ef4fcdf5a	build(deps-dev): bump @typescript-eslint/eslint-plugin (#2010 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.7.0 to 5.8.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.8.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	eae68a20b9	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#2012 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.60 to 5.0.0-alpha.61. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
dependabot[bot]	992114f0ab	build(deps): bump @types/react in /app/vmui/packages/vmui (#2011 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.37 to 17.0.38. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-01-03 16:57:14 +02:00
Dima Lazerka	55b4436804	Add check-rebased Github action (#2002 ) It will prevent merging in a branch that's not based on its base branch HEAD, leading to streamlined history. Note it will not prevent squash commits, nor commits directly to base branch.	2022-01-03 16:57:10 +02:00
Yury Molodov	85dc41007b	vmui: legend fixes (#1995 ) * feat: add a reset query by clicking the logo * feat: add sequence number for query fields * feat: invert behavior on the graph's legend * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-23 12:15:21 +02:00
Yurii Kravets	f6e2e0dbd1	Update README.md (#1996 ) * Update README.md go 1.16 -> 1.17 * Update README.md * Update README.md * Update Cluster-VictoriaMetrics.md * Update Single-server-VictoriaMetrics.md * Update vmauth.md * Update vmbackup.md * Update vmrestore.md * Update vmagent.md * Update vmctl.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md	2021-12-23 12:11:30 +02:00
Aliaksandr Valialkin	b18d80c247	docs/CHANGELOG.md: document `543bd0ea0c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999	2021-12-23 12:08:41 +02:00
Roman Khavronenko	d1c474c5b3	vmselect: update /query_exemplars placeholder (#2000 ) Grafana expects `data` in response to be a slice and logs an err if it is not: ``` err="[]v1.ExemplarQueryResult: decode slice: expect [ or n, but found , error found in #0 byte of ...\|\|..., bigger context ...\|\|..." ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-23 12:08:40 +02:00
Aliaksandr Valialkin	c1722003a2	lib/promscrape: scrape replicated targets at different offsets in vmagent replicated clustering mode This guarantees that the deduplication consistently leaves samples from the same vmagent replica. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets	2021-12-23 00:21:41 +02:00
Aliaksandr Valialkin	419ea376a6	docs/FAQ.md: link to managed VictoriaMetrics at AWS See https://docs.victoriametrics.com/FAQ.html#what-is-the-pricing-for-victoriametrics	2021-12-22 23:17:00 +02:00
Nikolay	6cdc934c3d	adds restore.lock (#1988 ) * adds restore.lock it must prevent from running storage after incomplete restore process https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 * return back flock file deletion * Apply suggestions from code review * wip * docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-22 13:10:56 +02:00
Aliaksandr Valialkin	fcab2fc716	vendor: `make vendor-update`	2021-12-22 12:39:24 +02:00
Aliaksandr Valialkin	c018f47e81	go.mod: update minimum Go version from Go 1.16 to Go 1.17 VictoriaMetrics code uses features from Go 1.17, so the minimum Go version must be increased from Go 1.16 to Go 1.17 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1987	2021-12-22 12:39:22 +02:00
Roman Khavronenko	4010f548b5	dashboards: migrate from old table panel in cluster dashboard (#1993 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-22 11:21:06 +02:00
Aliaksandr Valialkin	566c9791be	Revert "Add .github/workflows/check-based-on-master (#1991 )" This reverts commit `06cf4e0f70`. This break merge requests to non-master branches - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1993#issuecomment-999403963	2021-12-22 11:20:20 +02:00
Dima Lazerka	7462ff3e8c	Add .github/workflows/check-based-on-master (#1991 )	2021-12-21 20:31:12 +02:00
Roman Khavronenko	e92c987dc3	vmalert: check if remoteWrite is configured for replay mode (#1990 ) * vmalert: check if remoteWrite is configured for replay mode The purpose of `replay` mode is to backfill results of recording or alerting rules. So `remoteWrite.url` should be required. Otherwise, process can fail on attempt to send data. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update app/vmalert/main.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 20:31:10 +02:00
Yury Molodov	0952caca24	vmui: add custom start range (#1989 ) * feat: add custom start range * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 20:22:48 +02:00
Aliaksandr Valialkin	727797a6fd	all: use logger.WithThrottler() where appropriate	2021-12-21 17:10:54 +02:00
Aliaksandr Valialkin	4dbf12254d	lib/promscrape: take into account the original job_name when creating an unique key per each scrape target This should handle the case when the original job_name has been changed in -promscrape.config , while the resulting job label remains the same because it is overriden via relabeling.	2021-12-21 16:42:42 +02:00
Aliaksandr Valialkin	c61aa01c71	docs/CHANGELOG.md: document `34fdc8881b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911	2021-12-21 16:42:38 +02:00
Roman Khavronenko	23e1de06ee	vmagent: add error log for skipped data block when rejected by receiv… (#1956 ) * vmagent: add error log for skipped data block when rejected by receiving side Previously, rejected data blocks were silently dropped - only metrics were update. From operational perspective, having an additional logging for such cases is preferable. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: throttle log messages about skipped blocks The new type of logger was added to logger pacakge. This new type supposed to control number of logged messages by time. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/logger: make LogThrottler public, so its methods can be inspected by external packages Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 16:42:38 +02:00
Denys Holius	e24f7884e2	bump revision of dashboards to latest (#1986 )	2021-12-21 12:11:52 +02:00
Denys Holius	4cb6334ff6	added packer build for DigitalOcean Droplets (#1917 ) * added packer build for DigitalOcean Droplets * fixed typo * added packer RELEASE_GUIDE.md, Makefile * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * added corrections amd improvements * added packer link & templating for sed version * fixed typo Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2021-12-21 12:10:08 +02:00
Aliaksandr Valialkin	41c2657f23	docs/Single-server-VictoriaMetrics.md: mention that recording rules in vmalert can be used for reducing the number of time series	2021-12-20 20:04:21 +02:00
Aliaksandr Valialkin	28a0d3faaa	docs/CHANGELOG.md: cut v1.71.0	2021-12-20 19:10:38 +02:00
Aliaksandr Valialkin	053e85ff3d	all: typo fix: unexected -> unexpected	2021-12-20 17:40:13 +02:00
Yury Molodov	4c9e563c89	vmui: graph fixes (#1982 ) * fix: remove disabling custom step when zooming * feat: add a dynamic calc of the width of the graph * fix: add validate y-axis limits * fix: correct axis limits for value 0 * fix: change logic create time series * fix: change types for tooltip * fix: correct points on the line * fix: change the logic for set graph width * fix: stop checking the period when auto-refresh is enabled * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-20 17:37:45 +02:00
Roman Khavronenko	89facbc5c4	dashboards/vmagent: fix cached datasource uid (#1984 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-20 17:34:55 +02:00
Roman Khavronenko	0311d3cc89	Dashboards cluster (#1983 ) * dashboards/cluster: add panels for vmstorage in read-only mode vmstorage readonly status panel was addded to "vmstorage" row. A one more panel for showing vminsert->vmstorage readonly status was added to troubleshooting row. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/cluster: add "Cache usage" panel The new panel supposed to show the % of the used cache compared to allowed size by type. It should help to determine underutilized types of caches. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/cluster: add "Merges deferred" panel The new panel supposed to show if there were deferred merges due to insufficient disk space. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/cluster: update Network panel for vminsert * delete bytes_written query, since in most cases it is insiginificant * change display type to Stack Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/cluster: bump version requirement Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-20 17:32:05 +02:00
Aliaksandr Valialkin	406cb06f8c	lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981	2021-12-20 17:26:07 +02:00
Aliaksandr Valialkin	996c3b0957	docs: mention that downsampling can be evaluated for free by running enterprise binaries	2021-12-20 15:57:52 +02:00
Aliaksandr Valialkin	b393a16a25	app/vmselect/vmui: `make vmui-update`	2021-12-20 13:50:55 +02:00
Yurii Kravets	3638e096f3	Update FAQ.md (#1765 ) * Update FAQ.md Adding explanation "Why do same metrics have differences in VictoriaMetrics and Prometheus dashboards?" * Update FAQ * Update FAQ.md * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2021-12-20 13:43:40 +02:00
dependabot[bot]	dc81e9d352	build(deps): bump react-scripts in /app/vmui/packages/vmui (#1977 ) Bumps [react-scripts](https://github.com/facebook/create-react-app/tree/HEAD/packages/react-scripts) from 4.0.3 to 5.0.0. - [Release notes](https://github.com/facebook/create-react-app/releases) - [Changelog](https://github.com/facebook/create-react-app/blob/main/CHANGELOG.md) - [Commits](https://github.com/facebook/create-react-app/commits/react-scripts@5.0.0/packages/react-scripts) --- updated-dependencies: - dependency-name: react-scripts dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:20:30 +02:00
dependabot[bot]	ea625f817c	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#1978 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.2.1 to 5.2.4. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.4/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:18:57 +02:00
dependabot[bot]	0ab40c940c	build(deps): bump @types/node in /app/vmui/packages/vmui (#1979 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.12 to 17.0.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:18:57 +02:00
dependabot[bot]	6abe210683	build(deps): bump @mui/material in /app/vmui/packages/vmui (#1976 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.3 to 5.2.4. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.4/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:18:57 +02:00
dependabot[bot]	9c8c775492	build(deps-dev): bump @babel/plugin-proposal-nullish-coalescing-operator (#1975 ) Bumps [@babel/plugin-proposal-nullish-coalescing-operator](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-proposal-nullish-coalescing-operator) from 7.16.0 to 7.16.5. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.16.5/packages/babel-plugin-proposal-nullish-coalescing-operator) --- updated-dependencies: - dependency-name: "@babel/plugin-proposal-nullish-coalescing-operator" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:18:57 +02:00
dependabot[bot]	5a367951f0	build(deps): bump @codemirror/basic-setup in /app/vmui/packages/vmui (#1974 ) Bumps [@codemirror/basic-setup](https://github.com/codemirror/basic-setup) from 0.19.0 to 0.19.1. - [Release notes](https://github.com/codemirror/basic-setup/releases) - [Changelog](https://github.com/codemirror/basic-setup/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/basic-setup/compare/0.19.0...0.19.1) --- updated-dependencies: - dependency-name: "@codemirror/basic-setup" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 13:18:57 +02:00
Aliaksandr Valialkin	3396f08874	app/vmselect/promql: follow-up after `177e345d8a` * Document changes_prometheus(), increase_prometheus() and delta_prometheus() functions. * Simplify their implementation * Mention these functions in docs/CHANGELOG.md	2021-12-20 13:15:06 +02:00
匠心零度	cc4725843b	add Prometheus semantics function :changes_prometheus、delta_prometheus、increase_prometheus (#1972 ) Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-12-20 13:15:05 +02:00
Aliaksandr Valialkin	b1196e594b	docs/FAQ.md: describe main reasons for high churn rate	2021-12-20 12:30:42 +02:00
Roman Khavronenko	ada18cd963	Dashboards vmagent updates (#1973 ) * dashboards/vmagent: shuffle panels for better visibility More important error/dropped panels were moved higher on the main row. Network usage panel moved to Resource usage row. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/vmagent: add Troubleshooting row to show top 5 instances/jobs by churn rate New panels are supposed to show top 5 jobs or targets which generate the most of the churn rate. They were placed into a new row "Troubleshooting". Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/vmagent: add panels for showing persistent queue saturation New panels were added to Torubleshooting row to show the persistent queue saturation. The corresponding alerts were added and linked to these panels as well. Signed-off-by: hagen1778 <roman@victoriametrics.com> * dashboards/vmagent: add alert "RejectedRemoteWriteDataBlocksAreDropped" New alert suppose to send a notification when vmagent starts to drop data blocks rejected by configured remote write destiantion. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-20 12:19:17 +02:00
dependabot[bot]	3ebdaec9b7	build(deps-dev): bump @typescript-eslint/eslint-plugin (#1968 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.7.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 12:19:13 +02:00
dependabot[bot]	205cc48453	build(deps): bump uplot from 1.6.17 to 1.6.18 in /app/vmui/packages/vmui (#1967 ) Bumps [uplot](https://github.com/leeoniya/uPlot) from 1.6.17 to 1.6.18. - [Release notes](https://github.com/leeoniya/uPlot/releases) - [Commits](https://github.com/leeoniya/uPlot/compare/1.6.17...1.6.18) --- updated-dependencies: - dependency-name: uplot dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 12:19:13 +02:00
Aliaksandr Valialkin	dfbd4ca2ab	vendor: `make vendor-update`	2021-12-20 12:12:14 +02:00
dependabot[bot]	f09ed020ed	build(deps-dev): bump @typescript-eslint/parser (#1965 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.7.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 12:12:10 +02:00
dependabot[bot]	32d2c5ce4b	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#1966 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.59 to 5.0.0-alpha.60. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 12:12:10 +02:00
dependabot[bot]	f32bc714d3	build(deps): bump typescript in /app/vmui/packages/vmui (#1969 ) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.5.3 to 4.5.4. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](https://github.com/Microsoft/TypeScript/compare/v4.5.3...v4.5.4) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-20 12:12:10 +02:00
John Seekins	fcb759ebc3	OpenTSDB Migration Fix (#1946 ) * Simplify queries to OpenTSDB (and make them properly appear in OpenTSDB query stats) and also tweak defaults a bit Signed-off-by: John Seekins <jseekins@datto.com>	2021-12-20 12:12:10 +02:00
Aliaksandr Valialkin	c446858d08	docs/CHANGELOG.md: document `6814cc6809` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943	2021-12-17 20:17:09 +02:00
Roman Khavronenko	1f0301c809	vmalert: always convert `step` value to seconds for better compatibility (#1955 ) When using `vmalert` with older Prometheus versions, the passed `step=2m` may be parsed by Prometheus with an err: "cannot parse \"2m0s\" to a valid duration". In order to improve compatibility vmalert will always convert step duration to seconds. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-17 20:17:09 +02:00
Aliaksandr Valialkin	f22aab411b	lib/storage: properly update per-part `min_dedup_interval` file contents after merge Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value This is a follow-up for `4ff647137a`	2021-12-17 20:12:18 +02:00
Aliaksandr Valialkin	c2b0a6109b	app/vmselect/prometheus: typo fix after bf0e3ad2171219b7883b149999c4dc0ac15b9797	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	7c3239966e	app/vmselect/vmui: `make vmui-update`	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	d9f075622b	app/vmselect: de-duplicate data exported via `/api/v1/export/csv` by default Previously the exported data wasn't de-duplicated. Now it is possible to export the raw data without deduplication by passing reduce_mem_usage=1 query arg to /api/v1/export/csv See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837	2021-12-17 11:52:42 +02:00
Roman Khavronenko	e7cca46479	docs: add Benchmarks section (#1950 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-17 11:52:42 +02:00
Roman Khavronenko	a92f53486a	docs: update `MetricsQL`.`Subquery` section description (#1951 ) * simplify sentences; * fix typo. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-17 11:52:42 +02:00
Yury Molodov	7d32ca9ed3	fix: return query for app mode (#1954 )	2021-12-17 11:52:42 +02:00
Aliaksandr Valialkin	5bd4e47a9e	lib/promscrape: allow up to 5 redirects when scraping a target by default See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:45 +02:00
Aliaksandr Valialkin	c170841951	app/vmselect/promql: add `timestamp_with_name(m[d])` function This function works the same as `timestamp()`, but doesn't remove source time series names. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388	2021-12-15 23:38:09 +02:00
Aliaksandr Valialkin	5b4cb65138	docs: mention `-storage.minFreeDiskSpaceBytes` command-line flag at `capacity planning` section See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-12-15 21:43:21 +02:00
Yury Molodov	603be150c4	vmui: introduce application mode (#1949 ) * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * feat: introduce application mode * build vmui * Revert "build vmui" This reverts commit `c0e2415550`. * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-15 21:34:04 +02:00
Aliaksandr Valialkin	946769a828	docs/Cluster-VictoriaMetrics.md: mention about the added downsampling support	2021-12-15 16:39:35 +02:00
Aliaksandr Valialkin	3b8a300588	docs: document the added dowsnampling support in VictoriaMetrics enterprise	2021-12-15 16:25:32 +02:00
Aliaksandr Valialkin	d36fdbe537	lib/storage: deduplicate samples more thoroughly Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in higher disk space usage.	2021-12-15 16:00:30 +02:00
Aliaksandr Valialkin	bc3923111b	lib/storage: return dedup interval in milliseconds from GetDedupInterval() This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin	68cafdcf0b	app/victoria-metrics: mention https://docs.victoriametrics.com/#downsampling in the description for -dedup.minScrapeInterval command-line flag	2021-12-15 13:18:45 +02:00
Aliaksandr Valialkin	cdfe854c9b	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:52:29 +02:00
Aliaksandr Valialkin	c922c7af9a	lib/storage: convert alternate regexps into Graphite wildcards inside `__graphite__` pseudo-label For example, `{__graphite__=~"foo.(bar\|baz)"}` is automatically converted to `{__graphite__=~"foo.{bar,baz}"}` before execution. This allows using multi-value Grafana template variables such as `{__graphite__=~"foo.($app)"}`.	2021-12-14 19:55:59 +02:00
Aliaksandr Valialkin	b73dc2b3df	deployment/docker/docker-compose.yml: update Grafana version from 8.2.2 to 8.3.2 See https://grafana.com/blog/2021/12/10/grafana-8.3.2-and-7.5.12-released-with-moderate-severity-security-fix/	2021-12-14 15:09:37 +02:00
Aliaksandr Valialkin	eedc2aac08	docs/CHANGELOG.md: link to the issue about `unaligned 64-bit atomic opertion` panic on 32-bit architectures	2021-12-14 15:00:46 +02:00
Aliaksandr Valialkin	d02553831d	docs/CHANGELOG.md: an attempt to properly show $labels.alertname at https://docs.victoriametrics.com/CHANGELOG.html	2021-12-14 14:57:03 +02:00
Yury Molodov	b3389ce26b	vmui: custom step (#1942 ) * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * feat: add custom step * update package-lock.json * docs: document that VMUI now supports overriding of `step` query arg, which is passed to `/api/v1/query_range` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-14 14:52:44 +02:00
Yury Molodov	28655e8eb8	vmui: minor fixes (#1936 ) * feat: add a label for the Query field * fix: change zoom position * fix: add description and error code to alerts * fix: correct logic query history * fix: correct update query history * app/vmselect/vmui: `make vmui-update` * docs/CHANGELOG.md: document bugfixes Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-13 13:44:32 +02:00
Aliaksandr Valialkin	3d86e6353b	app/vmselect/vmui: `make vmui-update`	2021-12-13 13:33:35 +02:00
Aliaksandr Valialkin	24d0bb7c2a	docs/Single-server-VictoriaMetrics.md: added a link to Graphite paths and wildcards - https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards	2021-12-13 13:30:02 +02:00
dependabot[bot]	1e9d4063e0	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#1935 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.58 to 5.0.0-alpha.59. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	72487ab68a	build(deps): bump @codemirror/commands in /app/vmui/packages/vmui (#1933 ) Bumps [@codemirror/commands](https://github.com/codemirror/commands) from 0.19.5 to 0.19.6. - [Release notes](https://github.com/codemirror/commands/releases) - [Changelog](https://github.com/codemirror/commands/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/commands/compare/0.19.5...0.19.6) --- updated-dependencies: - dependency-name: "@codemirror/commands" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	5f6d5ba42e	build(deps-dev): bump @typescript-eslint/parser (#1934 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.5.0 to 5.6.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.6.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	f2f6d1519a	build(deps): bump @codemirror/view in /app/vmui/packages/vmui (#1931 ) Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.26 to 0.19.29. - [Release notes](https://github.com/codemirror/view/releases) - [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/view/compare/0.19.26...0.19.29) --- updated-dependencies: - dependency-name: "@codemirror/view" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	1510808fbd	build(deps): bump @emotion/react in /app/vmui/packages/vmui (#1930 ) Bumps [@emotion/react](https://github.com/emotion-js/emotion) from 11.7.0 to 11.7.1. - [Release notes](https://github.com/emotion-js/emotion/releases) - [Changelog](https://github.com/emotion-js/emotion/blob/main/CHANGELOG.md) - [Commits](https://github.com/emotion-js/emotion/compare/@emotion/react@11.7.0...@emotion/react@11.7.1) --- updated-dependencies: - dependency-name: "@emotion/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	afc1061e87	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#1929 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.2.0 to 5.2.1. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.1/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	80baf9a41f	build(deps): bump @types/node in /app/vmui/packages/vmui (#1932 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.11 to 16.11.12. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	ef126e352f	build(deps): bump @mui/material in /app/vmui/packages/vmui (#1925 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.2 to 5.2.3. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.3/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	cda420ebb4	build(deps): bump @testing-library/jest-dom in /app/vmui/packages/vmui (#1924 ) Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.16.0 to 5.16.1. - [Release notes](https://github.com/testing-library/jest-dom/releases) - [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/jest-dom/compare/v5.16.0...v5.16.1) --- updated-dependencies: - dependency-name: "@testing-library/jest-dom" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	f749a2a7f7	build(deps): bump typescript in /app/vmui/packages/vmui (#1926 ) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.5.2 to 4.5.3. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](https://github.com/Microsoft/TypeScript/compare/v4.5.2...v4.5.3) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	fd8772d122	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#1927 ) Bumps [@mui/styles](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-styles) from 5.2.2 to 5.2.3. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.3/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
dependabot[bot]	b9a2900d9f	build(deps-dev): bump @typescript-eslint/eslint-plugin (#1928 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.5.0 to 5.6.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.6.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-13 13:29:54 +02:00
Aliaksandr Valialkin	7b38507fc1	docs/CHANGELOG.md: document `7c3b6365f0`	2021-12-12 19:10:58 +02:00
Yury Molodov	6367edae1d	vmui: add a label for the Query field (#1923 ) * feat: add a label for the Query field * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-12 19:10:56 +02:00
Aliaksandr Valialkin	945135620a	deployment/docker: update Go builder from v1.17.4 to v1.17.5 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.5+label%3ACherryPickApproved	2021-12-12 18:18:52 +02:00
Aliaksandr Valialkin	aa61a74c31	vendor: `make vendor-update`	2021-12-12 18:10:34 +02:00
Aliaksandr Valialkin	addae7fc6a	app/vmselect/promql: preserve the order of time series passed to `limit_offset()` function Previously time series passed to `limit_offset()` were shuffled according to hash for their labels. This was unexpected behaviour for most users. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951	2021-12-12 18:07:15 +02:00
Aliaksandr Valialkin	988003536d	docs/CHANGELOG.md: document `2851709745` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921	2021-12-10 12:14:18 +02:00
Roman Khavronenko	e7d7f6c94e	vmalert: update the order of service labels attaching (#1922 ) Service labels like `alertname` or `alertgroup` were attached after template expanding for `labels` section. Because of this, labels `alertname` or `alertgroup` weren't available for templating in `labels` section of alert's definition. This commit changes the order of labels attaching and adds a test for verifying these labels availability. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-10 12:14:16 +02:00
Aliaksandr Valialkin	78656d91cd	app/vmui/README.md: remove `features` chapter, since it lists unimportant and/or misleading features The main user-visible features for vmui are documented at https://docs.victoriametrics.com/#vmui .	2021-12-09 19:51:33 +02:00
Aliaksandr Valialkin	e267355706	docs/Single-server-VictoriaMetrics.md: add a link to https://github.com/denisgolius/victoriametrics-ru-links	2021-12-09 19:43:28 +02:00
Aliaksandr Valialkin	3366124341	deployment/docker: update Go builder from v1.17.3 to v1.17.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.4+label%3ACherryPickApproved	2021-12-09 18:51:50 +02:00
Aliaksandr Valialkin	d91adcba3c	docs: document the ability to investigate correlation between two queries at /vmui This is a follow-up for `c1fd93e8a0`	2021-12-08 17:24:55 +02:00
Yury Molodov	39e29a50f1	vmui: multiple queries (#1916 ) * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * feat: remove save the last query to local storage * fix: handle an error in a table with subqueries * feat: store display type in URL * Revert "feat: store display type in URL" This reverts commit `ccc242c69a`. * feat: store display type in URL * refactor: move the time setting to a folder * refactor: move the query configurator to a folder * refactor: move the auth settings to a folder * feat: improve styles * feat: add multi query * update package-lock * feat: add display multiple queries * feat: add limits for multiple queries * update dependencies * feat: add history for multiple queries * feat: add line type to legend * feat: change style for switch * feat: change the logic for axes limits for multiple queries * update package-lock.json * update dependencies * feat: add the filter to legend * wip * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. * lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs. Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-08 16:43:18 +02:00
Aliaksandr Valialkin	38f5bc7451	lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs.	2021-12-08 16:15:12 +02:00
Aliaksandr Valialkin	ddd2f531bc	docs/Single-server-VictoriaMetrics.md: add LinkedIn public channel	2021-12-08 13:16:10 +02:00
Aliaksandr Valialkin	e21d1353f4	docs/Release-Guide.md: refresh the list of channels for publishing release notes	2021-12-08 13:16:09 +02:00
Aliaksandr Valialkin	7d8c481960	app/vmalert/config: sort extra_filter labels before passing them to query args in order to get consistent order of query args across runs This fixes TestGroupParams test - see https://github.com/VictoriaMetrics/VictoriaMetrics/runs/4432510244?check_suite_focus=true#step:5:288	2021-12-08 13:04:08 +02:00
Aliaksandr Valialkin	1c09881cf7	docs/Single-server-VictoriaMetrics.md: move features chapter above the `case studies` chapter	2021-12-08 12:49:50 +02:00
Aliaksandr Valialkin	54d4a1c959	app/vmselect: accept optional `extra_filters[]` query args for all the supported Prometheus querying APIs See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863	2021-12-06 17:33:49 +02:00
Aliaksandr Valialkin	9aa9b081a4	app/vminsert: add `-maxLabelValueLen` command-line flag See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908	2021-12-06 11:42:24 +02:00
dependabot[bot]	dc00e95010	build(deps-dev): bump @typescript-eslint/eslint-plugin (#1902 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.4.0 to 5.5.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.5.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-06 10:20:05 +02:00
Aliaksandr Valialkin	df810cfc9a	app/vmselect/vmui: `make vmui-update`	2021-12-06 10:18:03 +02:00
Aliaksandr Valialkin	6af4c0ca95	vendor: `make vendor-update`	2021-12-06 09:59:06 +02:00
dependabot[bot]	d919c16419	build(deps): bump @codemirror/view in /app/vmui/packages/vmui (#1901 ) Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.21 to 0.19.26. - [Release notes](https://github.com/codemirror/view/releases) - [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/view/compare/0.19.21...0.19.26) --- updated-dependencies: - dependency-name: "@codemirror/view" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-06 09:59:02 +02:00
dependabot[bot]	2a28ce9fcb	build(deps): bump @types/node in /app/vmui/packages/vmui (#1903 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.10 to 16.11.11. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-06 09:59:02 +02:00
dependabot[bot]	8da2e4bd20	build(deps): bump @testing-library/jest-dom in /app/vmui/packages/vmui (#1904 ) Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.15.1 to 5.16.0. - [Release notes](https://github.com/testing-library/jest-dom/releases) - [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/jest-dom/compare/v5.15.1...v5.16.0) --- updated-dependencies: - dependency-name: "@testing-library/jest-dom" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-06 09:59:02 +02:00
dependabot[bot]	881c178e48	build(deps-dev): bump @typescript-eslint/parser (#1905 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.4.0 to 5.5.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.5.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-12-06 09:59:02 +02:00
Aliaksandr Valialkin	51f2eb3c46	lib/workingsetcache: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures The panic has been introduced in `7275ebf91a`	2021-12-03 01:22:30 +02:00
Aliaksandr Valialkin	d40441947a	app: allow specifying http and https urls in the following command-line flags * -promscrape.config * -relabelConfig * -remoteWrite.relabelConfig * -remoteWrite.urlRelabelConfig	2021-12-03 00:11:47 +02:00
Aliaksandr Valialkin	daaea1eb2c	app/vmauth: follow-up for `13368bed18` * Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md * Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file should be read from the given path. For example, in `-promscrape.config` at `vmagent`.	2021-12-02 23:34:15 +02:00
Tiago Magalhães	fd3c9730b0	vmauth: support for reading remote auth config file (#1898 ) * add support for reading remote auth_config file via http * fix lint * fix defer on close body Co-authored-by: Tiago Magalhães <tmagalhaes@wavecom.pt>	2021-12-02 23:34:13 +02:00
Aliaksandr Valialkin	0f9aaf9cec	docs/CHANGELOG.md: cut v1.70.0	2021-12-02 15:01:39 +02:00
Aliaksandr Valialkin	8abf415225	docs/CHANGELOG.md: document `0afd14a14a`	2021-12-02 14:51:57 +02:00
Roman Khavronenko	582c063698	vmalert: introduce additional HTTP URL params per-group configuration (#1892 ) * vmalert: introduce additional HTTP URL params per-group configuration The new group field `params` allows to configure custom HTTP URL params per each group. These params will be applied to every request before executing rule's expression. Hot config reload is also supported. Field `extra_filter_labels` was deprecated in favour of `params` field. vmalert will print deprecation log message if config file contains the deprecated field. `params` fields are supported by both Prometheus and Graphite datasource types. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: provide more examples for `params` field Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: set higher priority for `params` setting If there would be a conflict between URL params set in `datasource.url` flag and params in group definition the latter will have higher priority. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-02 14:51:54 +02:00
Roman Khavronenko	8b67168609	ci: bump go version to 1.17 (#1895 ) The bump was required for `vmalert` package. `vmalert` docs now also contain an updated description. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-02 14:51:45 +02:00
Aliaksandr Valialkin	b885a3b6e9	lib/httpserver: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does This improves integration with third-party solutions, which rely on these endpoints. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833	2021-12-02 14:37:50 +02:00
Aliaksandr Valialkin	c540235470	app: use relative paths instead of absolute paths for the supported http handlers on the main page This allows hiding VictoriaMetrics components behind proxies, which serve pages at different path prefixes See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858	2021-12-02 13:54:15 +02:00
Aliaksandr Valialkin	d1289383eb	lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp	2021-12-02 13:44:01 +02:00
Aliaksandr Valialkin	37a2bea072	lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865	2021-12-02 13:33:50 +02:00
Aliaksandr Valialkin	ca9827fecc	docs/FAQ.md: mention that VictoriaMetrics can be queried via vmui	2021-12-02 13:16:34 +02:00
Aliaksandr Valialkin	246417601c	vendor: `make vendor-update`	2021-12-02 12:38:38 +02:00
Aliaksandr Valialkin	2b7dee15dd	app/{vmbackup,vmrestore}: export internal metrics at `/metrics` http handler	2021-12-02 11:56:34 +02:00
Roman Khavronenko	fc4e59ec1d	vmalert: adjust topologies docs in README (#1893 ) Commit changes images width and order in topologies section for better readability. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-12-02 10:33:47 +02:00
Denys Holius	60a83fefbc	added guide-vmcluster-multiple-retention-setup to guides list (#1891 )	2021-12-02 10:33:47 +02:00
Roman Khavronenko	6823818843	Vmalert docs upd (#1890 ) * vmalert: add topology examples in docs * vmalert: docs typo fix	2021-12-02 10:33:47 +02:00
Aliaksandr Valialkin	ab4be24397	app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query: vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9	2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin	d4655beae8	lib/fs: add `vm_filestream_read_duration_seconds_total` and `vm_filestream_write_duration_seconds_total` metrics These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`	2021-12-02 09:13:20 +02:00
Aliaksandr Valialkin	22e7b3c95a	README.md: add a link to https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html for multi-retention setups	2021-12-01 12:49:05 +02:00
Yurii Kravets	8e4aa91475	Create guide-vmcluster-multiple-retention-setup.md (#1888 ) * Create guide-vmcluster-multiple-retention-setup.md * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-01 12:42:56 +02:00
Yurii Kravets	38c9981a98	Add retention-scheme.png	2021-12-01 12:30:06 +02:00
Aliaksandr Valialkin	d0c0c599a5	docs/CHANGELOG.md: document `06eff5a72c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487	2021-12-01 12:30:03 +02:00
Aliaksandr Valialkin	2e43cd9d62	lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges	2021-12-01 12:30:03 +02:00
Nikolay	cf1d2f289b	removes FileSize from backup part key (#1872 ) * removes FileSize from backup part key it should fix download restoration for backups * Update lib/backup/common/part.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-01 12:30:03 +02:00
Aliaksandr Valialkin	bf6eb624f0	docs/Articles.md: added a link to https://mist.io/blog/2021-11-26-kubernetes-and-victoriametrics-in-Mist-v4-6	2021-11-30 15:00:05 +02:00
Aliaksandr Valialkin	5af9480e8e	docs/Single-server-VictoriaMetrics.md: document that the deduplication is applied only when exporting data in JSON line format The exported data isn't de-duplicated by default due to performance reasons. It is expected that the de-duplication is applied during importing the exported data. The deduplication is applied only when exporting data via /api/v1/export if `reduce_mem_usage=1` query arg isn't passed to the request. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837	2021-11-30 13:06:50 +02:00
Aliaksandr Valialkin	71c0f7cce3	lib/storage: take into account `-storage.minFreeDiskSpaceBytes` when performing big merges Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-11-30 12:56:53 +02:00
Aliaksandr Valialkin	120fedf1ac	app/vmselect/vmui: `make vmui-update`	2021-11-30 01:38:59 +02:00
Roman Khavronenko	83c46eaf1a	vmalert: continue to print errors for bad config during hot reload (#1871 ) Previously, vmalert would print an err message and set vmalert_config_last_reload_successful=0 only once during a hot reload of a bad config. Such behaviour may result into non noticed event of a bad config reload attempt Now, it continues to print error messages and keep vmalert_config_last_reload_successful state until successful attempt will be made or config state will be rolled back to prev state. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-11-30 01:24:20 +02:00
Roman Khavronenko	48fb972ddc	vmalert: make notifier.Addr optional (#1870 ) For a long time notifier.Addr flag was required. The assumption was that vmalert will be always used for alerting. However, practice shows that some users need only recording rules. In this case, requirement of notifier.Addr is ambigious. The change verifies if loaded config contains recording or alerting rules and if there are corresponding flags set. This is true for initial config load and hot reload. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-11-30 01:23:00 +02:00
Aliaksandr Valialkin	7b88daf2ec	docs/CHANGELOG.md: document `852a895b70`	2021-11-30 01:22:59 +02:00
Aliaksandr Valialkin	fdca152322	docs/CHANGELOG.md: document `f05cddd2fc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1830	2021-11-30 01:16:16 +02:00
guidao	6fa7ad69fc	fix #1830 (#1861 ) Co-authored-by: wangfeng <wangfeng@zhihu.com>	2021-11-30 01:16:15 +02:00
Aliaksandr Valialkin	cc773ac199	docs/CHANGELOG.md: document `9dd650f67f`	2021-11-30 01:09:30 +02:00
Yury Molodov	acc05e16c1	feat: store display type in URL (#1855 )	2021-11-30 01:09:28 +02:00
Aliaksandr Valialkin	b7eb4bb6cb	docs/CHANGELOG.md: document `624ad73705`	2021-11-30 01:05:29 +02:00
Yury Molodov	0108c9d2cc	vmui: handle an error in a table with subqueries (#1854 ) * fix: handle an error in a table with subqueries * feat: store display type in URL * Revert "feat: store display type in URL" This reverts commit `ccc242c69a`.	2021-11-30 01:05:28 +02:00
Aliaksandr Valialkin	e4c329f5a8	docs/CHANGELOG.md: document `c6d5927281`	2021-11-30 01:00:56 +02:00
Yury Molodov	d0313b9401	feat: remove save the last query to local storage (#1853 )	2021-11-30 01:00:54 +02:00
Yury Molodov	500945499f	vmui: optimize render (#1852 ) * feat: change duration by "enter" * fix: optimize data processing for chart * feat: set minimum step to 1ms * update dependencies * update package-lock * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-30 00:57:35 +02:00
Aliaksandr Valialkin	975498d402	lib/protoparser/prometheus: follow-up for `8e338632a3` Do not spend CPU time on error message formatting if error logger is disabled	2021-11-30 00:51:15 +02:00
Nikolay	40f0726147	Changes unmarshallRow logger to noop for getRowsDiff (#1835 )	2021-11-30 00:51:14 +02:00
Aliaksandr Valialkin	648d788a45	app/vmselect/vmui: `make vmui-update`	2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin	4ad397188e	lib/protoparser: do not log `connection reset by peer` error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections This error is expected, so there is no need in spamming the log with this error.	2021-11-29 21:58:11 +02:00
dependabot[bot]	46a3b24162	build(deps): bump @types/jest in /app/vmui/packages/vmui (#1886 ) Bumps [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest) from 27.0.2 to 27.0.3. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest) --- updated-dependencies: - dependency-name: "@types/jest" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	009756e818	build(deps): bump @mui/material in /app/vmui/packages/vmui (#1885 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.2.1 to 5.2.2. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.2/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	67491739f1	build(deps): bump typescript in /app/vmui/packages/vmui (#1884 ) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.4.4 to 4.5.2. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](https://github.com/Microsoft/TypeScript/compare/v4.4.4...v4.5.2) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	00043deeb3	build(deps): bump @testing-library/jest-dom in /app/vmui/packages/vmui (#1883 ) Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.15.0 to 5.15.1. - [Release notes](https://github.com/testing-library/jest-dom/releases) - [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md) - [Commits](https://github.com/testing-library/jest-dom/compare/v5.15.0...v5.15.1) --- updated-dependencies: - dependency-name: "@testing-library/jest-dom" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	0983144c16	build(deps-dev): bump eslint-plugin-react in /app/vmui/packages/vmui (#1882 ) Bumps [eslint-plugin-react](https://github.com/yannickcr/eslint-plugin-react) from 7.26.1 to 7.27.1. - [Release notes](https://github.com/yannickcr/eslint-plugin-react/releases) - [Changelog](https://github.com/yannickcr/eslint-plugin-react/blob/master/CHANGELOG.md) - [Commits](https://github.com/yannickcr/eslint-plugin-react/compare/v7.26.1...v7.27.1) --- updated-dependencies: - dependency-name: eslint-plugin-react dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	79274cd218	build(deps): bump @codemirror/state in /app/vmui/packages/vmui (#1881 ) Bumps [@codemirror/state](https://github.com/codemirror/state) from 0.19.4 to 0.19.6. - [Release notes](https://github.com/codemirror/state/releases) - [Changelog](https://github.com/codemirror/state/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/state/compare/0.19.4...0.19.6) --- updated-dependencies: - dependency-name: "@codemirror/state" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	d19cc19efc	build(deps): bump @mui/icons-material in /app/vmui/packages/vmui (#1880 ) Bumps [@mui/icons-material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-icons-material) from 5.0.5 to 5.2.0. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.0/packages/mui-icons-material) --- updated-dependencies: - dependency-name: "@mui/icons-material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	010b274e90	build(deps): bump @types/react in /app/vmui/packages/vmui (#1879 ) Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.34 to 17.0.37. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react) --- updated-dependencies: - dependency-name: "@types/react" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	4c1fa3f32d	build(deps): bump @codemirror/view in /app/vmui/packages/vmui (#1878 ) Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.20 to 0.19.21. - [Release notes](https://github.com/codemirror/view/releases) - [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/view/compare/0.19.20...0.19.21) --- updated-dependencies: - dependency-name: "@codemirror/view" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	bcb973ba4b	build(deps): bump @emotion/react in /app/vmui/packages/vmui (#1877 ) Bumps [@emotion/react](https://github.com/emotion-js/emotion) from 11.6.0 to 11.7.0. - [Release notes](https://github.com/emotion-js/emotion/releases) - [Changelog](https://github.com/emotion-js/emotion/blob/main/CHANGELOG.md) - [Commits](https://github.com/emotion-js/emotion/compare/@emotion/react@11.6.0...@emotion/react@11.7.0) --- updated-dependencies: - dependency-name: "@emotion/react" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	34d642abed	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#1876 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.55 to 5.0.0-alpha.58. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	959ae64594	build(deps): bump @emotion/styled in /app/vmui/packages/vmui (#1814 ) Bumps [@emotion/styled](https://github.com/emotion-js/emotion) from 11.3.0 to 11.6.0. - [Release notes](https://github.com/emotion-js/emotion/releases) - [Changelog](https://github.com/emotion-js/emotion/blob/main/CHANGELOG.md) - [Commits](https://github.com/emotion-js/emotion/compare/@emotion/styled@11.3.0...@emotion/styled@11.6.0) --- updated-dependencies: - dependency-name: "@emotion/styled" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	68bfa18a7a	build(deps): bump uplot from 1.6.16 to 1.6.17 in /app/vmui/packages/vmui (#1848 ) Bumps [uplot](https://github.com/leeoniya/uPlot) from 1.6.16 to 1.6.17. - [Release notes](https://github.com/leeoniya/uPlot/releases) - [Commits](https://github.com/leeoniya/uPlot/compare/1.6.16...1.6.17) --- updated-dependencies: - dependency-name: uplot dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	1a5174ac2d	build(deps): bump @mui/styles in /app/vmui/packages/vmui (#1875 ) Bumps [@mui/styles](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-styles) from 5.0.2 to 5.2.2. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.2/packages/mui-styles) --- updated-dependencies: - dependency-name: "@mui/styles" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	314a7d1690	build(deps): bump @codemirror/autocomplete in /app/vmui/packages/vmui (#1874 ) Bumps [@codemirror/autocomplete](https://github.com/codemirror/autocomplete) from 0.19.4 to 0.19.9. - [Release notes](https://github.com/codemirror/autocomplete/releases) - [Changelog](https://github.com/codemirror/autocomplete/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/autocomplete/compare/0.19.4...0.19.9) --- updated-dependencies: - dependency-name: "@codemirror/autocomplete" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	8374c20017	build(deps): bump @types/node in /app/vmui/packages/vmui (#1868 ) Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.6 to 16.11.10. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
dependabot[bot]	de97640506	build(deps): bump @mui/material in /app/vmui/packages/vmui (#1866 ) Bumps [@mui/material](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-material) from 5.0.6 to 5.2.1. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/v5.2.1/packages/mui-material) --- updated-dependencies: - dependency-name: "@mui/material" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin	3d20d229d0	vendor: `make vendor-update`	2021-11-29 12:36:56 +02:00
Aliaksandr Valialkin	d5ae9e9619	app/vmselect/vmui: `make vmui-update`	2021-11-29 12:23:12 +02:00
Denis Golius	efdbe0adfe	Bumped Alpine linux version to 3.15.0	2021-11-29 12:18:04 +02:00
dependabot[bot]	d2a3946fe1	build(deps-dev): bump @typescript-eslint/parser (#1843 ) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 5.3.0 to 5.4.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.4.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 12:17:33 +02:00
dependabot[bot]	0a43058fe3	build(deps): bump @emotion/react in /app/vmui/packages/vmui (#1817 ) Bumps [@emotion/react](https://github.com/emotion-js/emotion) from 11.5.0 to 11.6.0. - [Release notes](https://github.com/emotion-js/emotion/releases) - [Changelog](https://github.com/emotion-js/emotion/blob/main/CHANGELOG.md) - [Commits](https://github.com/emotion-js/emotion/compare/@emotion/react@11.5.0...@emotion/react@11.6.0) --- updated-dependencies: - dependency-name: "@emotion/react" dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 12:16:59 +02:00
dependabot[bot]	73ae8420c7	build(deps): bump @codemirror/view in /app/vmui/packages/vmui (#1841 ) Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.14 to 0.19.20. - [Release notes](https://github.com/codemirror/view/releases) - [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md) - [Commits](https://github.com/codemirror/view/compare/0.19.14...0.19.20) --- updated-dependencies: - dependency-name: "@codemirror/view" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 12:16:20 +02:00
dependabot[bot]	b16ca07b0d	build(deps): bump @mui/lab in /app/vmui/packages/vmui (#1842 ) Bumps [@mui/lab](https://github.com/mui-org/material-ui/tree/HEAD/packages/mui-lab) from 5.0.0-alpha.53 to 5.0.0-alpha.55. - [Release notes](https://github.com/mui-org/material-ui/releases) - [Changelog](https://github.com/mui-org/material-ui/blob/master/CHANGELOG.md) - [Commits](https://github.com/mui-org/material-ui/commits/HEAD/packages/mui-lab) --- updated-dependencies: - dependency-name: "@mui/lab" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-29 12:15:49 +02:00
Michael Fuller	8a94dd36d1	vmselect: in promql evaluation, return bytes requested when rollup memory limiter is unable to satisfy the request (#1838 ) Co-authored-by: Michael Fuller <mfuller@digitalocean.com>	2021-11-29 12:14:55 +02:00
Aliaksandr Valialkin	4c466b27f1	docs/CHANGELOG.md: document `695cb617b2`	2021-11-29 12:13:39 +02:00
John Seekins	50c33b7265	Simplify queries to OpenTSDB for migration (#1809 ) * Simplify queries to OpenTSDB (and make them properly appear in OpenTSDB query stats) and also tweak defaults a bit Signed-off-by: John Seekins <jseekins@datto.com> * remove extraneous printlns Signed-off-by: John Seekins <jseekins@datto.com> * remove empty line Signed-off-by: John Seekins <jseekins@datto.com> * fix bug in offset calcuation and closer to working with simpler queries Signed-off-by: John Seekins <jseekins@datto.com> * fix boolean eval Signed-off-by: John Seekins <jseekins@datto.com> * fix casting and check for multiple series Signed-off-by: John Seekins <jseekins@datto.com>	2021-11-29 12:13:38 +02:00
Aliaksandr Valialkin	84decc4254	app/vmselect/promql: consistently return zero from `deriv(const)`	2021-11-17 18:01:58 +02:00
Aliaksandr Valialkin	e93f46187d	lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent	2021-11-17 16:42:12 +02:00
Aliaksandr Valialkin	aa6d11a33f	app/vmselect/promql: add `now()` function, which returns the current timestamp as a floating-point value in seconds	2021-11-17 16:42:11 +02:00
Aliaksandr Valialkin	ee64a88ba3	vendor: `make vendor-update`	2021-11-17 13:40:36 +02:00
Aliaksandr Valialkin	67676880b3	app/vmauth: accept requests with Basic Auth username which is equal to `bearer_token` value from the `-auth.config`	2021-11-17 13:31:52 +02:00
Aliaksandr Valialkin	5610207549	docs: document -s3ForcePathStyle command-line option This is a follow-up for `b72eed1f5e`	2021-11-17 01:10:24 +02:00
Lan	6662714c6c	Add flag of S3ForcePathStyle (#1802 )	2021-11-17 01:10:22 +02:00
vic	273fbc2e8a	Update Cluster-VictoriaMetrics.md (#1806 ) replicationFactor flag should be passed to vmselect instead of vminsert for improving query speed:)	2021-11-17 01:01:08 +02:00
Aliaksandr Valialkin	aa01bb9349	app/vmalert/README.md: sync with docs/vmalert.md This is a follow-up after `d8c70903ec`	2021-11-17 00:56:53 +02:00
Florian Klink	89650de497	docs/vmalert.md: document vmalert url flags a bit more cleanly (#1823 ) Describe remoteWrite.url is used to persist rules and alerts state info, and add an additional paragraph explaining the separation between -remoteRead.url and -datasource.url. Fixes #1810.	2021-11-17 00:56:52 +02:00
Aliaksandr Valialkin	71082c7df8	app/vmauth: add ability to override the `username` label value for `vmauth_user_requests_total` metric by specifying `name` option in `-auth.config` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1805	2021-11-17 00:48:14 +02:00
Aliaksandr Valialkin	8fabca9f5d	docs/Single-server-VictoriaMetrics.md: add a link to vmalert rules backfilling at `Backfilling` chapter	2021-11-17 00:24:05 +02:00
Aliaksandr Valialkin	f1c0e9db25	docs/CHANGELOG.md: document the addition of `vm_tenant_used_tenant_bytes` metric, which shows the per-tenant disk usage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605	2021-11-16 23:35:48 +02:00
Aliaksandr Valialkin	c6fa2eaa3b	vendor: `make vendor-update`	2021-11-14 14:05:11 +02:00
Denys Holius	64af0de0fc	Bumped Alpine linux version to the latest (#1811 ) See this https://alpinelinux.org/posts/Alpine-3.14.3-released.html	2021-11-14 13:56:07 +02:00
Aliaksandr Valialkin	1eb756fd11	app/vmselect/promql: arrange function names in the code in alphabetical order This should simplify code maintenance in the future	2021-11-14 13:54:43 +02:00
Aliaksandr Valialkin	bdc813fc33	app/vmui/Dockerfile-web: update Go builder for vmui from v1.17.1 to v1.17.3	2021-11-14 13:54:41 +02:00
Aliaksandr Valialkin	8cb7074030	deployment/docker: allow using `/` chars in `ROOT_IMAGE` when running `make package-*` This fixes the following command: ROOT_IMAGE=gcr.io/distroless/static make package-victoria-metrics	2021-11-14 12:22:22 +02:00
Aliaksandr Valialkin	471c0ddd88	docs/Articles.md: add a linkt to OSA Con talk about `how clickhouse inspired us to build victoriametrics`	2021-11-12 14:21:42 +02:00
Aliaksandr Valialkin	82cf9f26db	app/vmselect/promql: prevent from incorrect calculations for `deriv()` over multiple samples with identical timestamps	2021-11-12 13:53:30 +02:00
Aliaksandr Valialkin	82917398b9	docs/CHANGELOG.md: document bugfixes in enteprise versions of vmagent and vmalert	2021-11-12 13:24:37 +02:00
Aliaksandr Valialkin	fd4d665079	docs/Single-server-VictoriaMetrics.md: mention that it is possible to send gzipped data to /api/v1/import/prometheus	2021-11-09 20:45:54 +02:00
Aliaksandr Valialkin	305507930c	app/vmauth: initialize reverse proxy only after flag.Parse() is called This should properly take into accoun the `-maxIdleConnsPerBackend` command-line flag value. Previously it was hardcoded to 100. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-11-09 19:23:44 +02:00
Aliaksandr Valialkin	4fb19fe34b	all: consistently return `application/json` content-type without `charset=utf-8` The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin	6346b78fa8	dashboards: consistently use regexp filters for template vars (#1799 ) Template vars may contain regexp when `all` is selected (.*) or when multiple values are selected (foo\|bar). So they must be passed to regexp filters.	2021-11-09 16:50:08 +02:00
Aliaksandr Valialkin	bc357d7132	docs/vmalert.md: improve wording in `Multitenancy` chapter	2021-11-09 14:20:30 +02:00
Aliaksandr Valialkin	31486f2244	app/vmselect/promql: properly return durations smaller than one second from duration_over_time() function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-09 11:42:43 +02:00
Aliaksandr Valialkin	57aaf914ac	docs: mention that graphs on the official dashboards contain useful hints	2021-11-08 19:54:25 +02:00
Aliaksandr Valialkin	a352b59b7e	docs/MetricsQL.md: mention than tlast_over_time() is an alias for timestamp()	2021-11-08 18:29:59 +02:00
Aliaksandr Valialkin	b1de180886	docs/CHANGELOG.md: cut v1.69.0	2021-11-08 15:47:48 +02:00
Yury Molodov	89e0f66bfc	vmui: migration MUI Core v4 to v5 (#1795 ) * migration MUI Core v4 to v5 * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-08 15:46:57 +02:00
Aliaksandr Valialkin	f41c02e475	lib/promscrape: improve logging for `scrape_config_files` parse errors Log the actual file path, which led to the parse error. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789	2021-11-08 13:34:26 +02:00
Aliaksandr Valialkin	3a1ac218ba	app/vmselect/promql: add `duration_over_time(m[d], max_interval)` function This function calculates the actual lifetime of the time series on the given lookbehdind window `d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-08 13:14:57 +02:00
Aliaksandr Valialkin	d59acb06c1	vendor: `make vendor-update`	2021-11-08 12:41:20 +02:00
Aliaksandr Valialkin	62edbcdb23	docs/MetricsQL.md: clarify documentation for `lifetime` function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780	2021-11-08 12:35:45 +02:00
Aliaksandr Valialkin	7cbb81c2a6	Makefile: add `TAG=v... make publish-release` rule for building and publishing a release for the given TAG	2021-11-08 12:29:24 +02:00
Aliaksandr Valialkin	2e237a6e8c	docs/CHANGELOG.md: document b9cdbcb5046315db96e1e7ca9923d09d0f30dc25	2021-11-08 12:13:32 +02:00
Yury Molodov	654686fd40	vmui: fix graph reset (#1788 ) * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * feat: change graph legend * update dependencies * update codemirror version * fix: correct update period time after zoom/pan * fix: optimize data processing for the graph * fix: eliminate memory leaks related to mouse events * fix: correct display of straight line * Merge branch 'master' into vmui-fix-reset-graph * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-05 20:58:49 +02:00
Aliaksandr Valialkin	e58630401e	app/{vmalert,vmbackup}/README.md: sync with docs after the commit `47d1612bf8`	2021-11-05 20:46:07 +02:00
João Paulo	5bd3518e4d	docs: fix multiple typos (#1787 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-11-05 20:46:06 +02:00
Aliaksandr Valialkin	9aa40fc0c3	docs/vmalert.md: document the addition of -defaultTenant.prometheus and -defaultTenant.graphite command-line options to enterprise version of vmalert	2021-11-05 20:04:53 +02:00
Aliaksandr Valialkin	56a25146cb	app/vmalert/datasource: use plain string literals instead of constants This removes the unneeded level of indirection and improves code readability. The "prometheus" and "graphite" constants aren't going to change in the future, so there is no sense in hiding them behind constants.	2021-11-05 19:58:15 +02:00
Aliaksandr Valialkin	31ac507bee	app/vmalert: remove `rule.type` config, since it doesnt play well with the upcoming default tenants for -clusterMode It is better from the consistency point of view to set up rule types at group level where tenant config is set up.	2021-11-05 19:52:41 +02:00
Aliaksandr Valialkin	e702b7a92f	app/vmagent: allow bigger number of in-memory blocks for big values of `-remoteWrite.queues` This should improve the maximum data ingestion speed for highly-loaded vmagent instances which run on beefy servers with many CPU cores and big amounts of RAM	2021-11-05 15:16:50 +02:00
Aliaksandr Valialkin	847004fa77	app/{vminsert,vmagent}: hide passwords and auth tokens by default at `/config` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin	bc72b83102	lib/promauth: do not show empty values in `oauth2` config section at `/config` page	2021-11-05 12:54:10 +02:00
Aliaksandr Valialkin	9806e4793f	deployment/dm: update Go builder from Go1.17.2 to Go1.17.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.3+label%3ACherryPickApproved	2021-11-05 12:50:34 +02:00
Denys Holius	a3182bc541	bumped grafana dashboards revisions for guides (#1784 )	2021-11-05 11:50:41 +02:00
Aliaksandr Valialkin	c8b9d9e96e	docs/FAQ.md: mention that VictoriaMetrics can be queried via Graphite API	2021-11-04 22:38:33 +02:00
Aliaksandr Valialkin	9edc12fa99	app/vmagent/remotewrite: fix parallel data sending to remote storage systems at `e0d2ba5608`	2021-11-04 16:58:53 +02:00
Aliaksandr Valialkin	63c29099ba	app/vmagent: add `-remoteWrite.maxRowsPerBlock` command-line option, which may be used for improving data ingestion performance under high load	2021-11-04 15:39:55 +02:00
Aliaksandr Valialkin	5f13ba8631	app/vmagent/remotewrite: send data to remote storage systems in parallel This should improve data ingestion speed many `-remoteWrite.url` options are configured	2021-11-04 15:04:02 +02:00
dependabot[bot]	08f487e7e8	build(deps-dev): bump @typescript-eslint/eslint-plugin (#183 ) Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 5.2.0 to 5.3.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/master/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v5.3.0/packages/eslint-plugin) --- updated-dependencies: - dependency-name: "@typescript-eslint/eslint-plugin" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-04 12:53:03 +02:00
Aliaksandr Valialkin	be9c7713d7	app/vmselect: `make vmui-update`	2021-11-04 12:16:55 +02:00
dependabot[bot]	c656e342dd	build(deps-dev): bump @babel/plugin-proposal-nullish-coalescing-operator (#1769 ) Bumps [@babel/plugin-proposal-nullish-coalescing-operator](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-proposal-nullish-coalescing-operator) from 7.14.5 to 7.16.0. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.16.0/packages/babel-plugin-proposal-nullish-coalescing-operator) --- updated-dependencies: - dependency-name: "@babel/plugin-proposal-nullish-coalescing-operator" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-04 12:16:55 +02:00
dependabot[bot]	bb8b426628	build(deps): bump @date-io/dayjs in /app/vmui/packages/vmui (#1770 ) Bumps [@date-io/dayjs](https://github.com/dmtrKovalenko/date-io) from 1.3.13 to 2.11.0. - [Release notes](https://github.com/dmtrKovalenko/date-io/releases) - [Commits](https://github.com/dmtrKovalenko/date-io/compare/v1.3.13...v2.11.0) --- updated-dependencies: - dependency-name: "@date-io/dayjs" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-11-04 12:16:55 +02:00
Denys Holius	2d1b24217a	bumped golangci-lint to the latest 1.43 (#1781 )	2021-11-04 12:16:55 +02:00
Roman Khavronenko	c78496d536	docs: make link to logos zip absolute (#1782 ) The relative link won't work for github-docs website, so we're changing it to absolute link. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-11-04 12:16:55 +02:00
Aliaksandr Valialkin	d445d22c0c	lib/promscrape: add `-promscrape.maxResponseHeadersSize` command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:27:55 +02:00
Aliaksandr Valialkin	a102fca4ac	app/vmselect/promql: add `limit_offset(limit, offset, q)` function, which can be used for paging over big number of time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778	2021-11-03 16:03:10 +02:00
Aliaksandr Valialkin	43bf0333eb	app/vmselect/promql: add `label_graphite_group()` function for extracting groups from Graphite metric names	2021-11-03 13:20:34 +02:00
Aliaksandr Valialkin	8a490bf375	docs/CHANGELOG.md: document `e3a91b186a`	2021-11-02 18:39:52 +02:00
Aliaksandr Valialkin	551a94ba7f	app/{vmagent,vminsert}: add ability to restrict access to /config page with authKey query arg The authKey can be configured via `-configAuthKey` command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-01 16:47:30 +02:00
Aliaksandr Valialkin	54f3f08874	vendor: `make vendor-update`	2021-11-01 16:08:01 +02:00
Aliaksandr Valialkin	d2b0c9556c	vendor: update github.com/VictoriaMetrics/metrics from v1.18.0 to v1.18.1	2021-11-01 16:08:00 +02:00
Aliaksandr Valialkin	7cf069ddfd	docs/Articles.md: add a link to https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986	2021-10-29 14:03:27 +03:00
Aliaksandr Valialkin	94d9c06fc1	docs/vmauth.md: typo fix	2021-10-28 14:06:28 +03:00
Aliaksandr Valialkin	6873d6d893	lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp	2021-10-28 12:48:34 +03:00
Aliaksandr Valialkin	105deb164c	lib/logger: show only explicitly set command-line flags in logs This reduces initial verbosity in logs	2021-10-28 11:03:21 +03:00
Aliaksandr Valialkin	38cbae006a	app/vmbackupmanager: fix links to images This is a follow-up after `bd6b8f7e31`	2021-10-27 21:36:25 +03:00
Yury Molodov	a16c88095c	vmui: correct migration material-ui (#1758 ) * migration material-ui * fix: rollback popover * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-27 20:43:24 +03:00
Aliaksandr Valialkin	b626d6d606	lib/promscrape: add `collapse` and `expand` buttons per each group of targets from the same scrape job	2021-10-27 20:04:03 +03:00
Aliaksandr Valialkin	2ebee4e741	app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules Previously only the lower part of 64-bit hash was used for calculating the offset. This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash for calculating the offset.	2021-10-27 20:04:02 +03:00
Aliaksandr Valialkin	92d01db85a	lib/protoparser/prometheus: optimize GetRowsDiff() function This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 , since the provided profile shows that the majority of CPU and memory is spent in this function during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.	2021-10-27 18:55:25 +03:00
Aliaksandr Valialkin	16f1aaf0b5	lib/protoparser/prometheus: add a benchmark for GetRowsDiff	2021-10-27 18:55:23 +03:00
Aliaksandr Valialkin	99784b21c1	all: fix build issues and tests for Apple M1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:07:19 +03:00
Aliaksandr Valialkin	6c1701d908	.github/workflows/main.yml: checkout code before installing dependencies Dependencies depend on Makefile rules from the code, so code checkout must run first	2021-10-26 22:09:38 +03:00
Aliaksandr Valialkin	f86dba1f5c	docs/CHANGELOG.md: mention the issue about missing `proxy_url` config option at `/config` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 22:07:07 +03:00
Aliaksandr Valialkin	d65912e47c	.github/workflows/main.yml: re-use makefile rules for installing goling, errcheck and golangci-lint	2021-10-26 22:05:09 +03:00
Aliaksandr Valialkin	ad445a06cd	lib/promscrape: properly show `proxy_url` option value at `/config` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin	aece35158d	vendor: `make vendor-update`	2021-10-26 19:38:27 +03:00
Yury Molodov	e7591577f0	vmui: update dependencies (#1754 ) * update dependencies * update codemirror version * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-26 19:32:31 +03:00
Aliaksandr Valialkin	eb783b0ef3	app/vmbackup/README.md: sync with docs/vmbackup.md after `e706fb5686`	2021-10-26 19:21:26 +03:00
Dima Lazerka	464f5d0910	Fix doc: vmbackup splits by 1 GiB not 100 MB (#1756 ) This is a follow-up for `bdd0a1cdb2`	2021-10-26 19:21:25 +03:00
Denys Holius	56970caded	fixed wrong path for npm dependabot checks (#1744 )	2021-10-26 19:16:35 +03:00
Aliaksandr Valialkin	9e8697c6c3	docs/CHANGELOG.md: document `3dbdf1632e`	2021-10-25 14:46:52 +03:00
Roman Khavronenko	5321127add	vmalert: allow groups with empty rules for compatibility reasons (#1742 ) Prometheus allows to have groups with no rules, so we should support it in vmalert as well for compatibility reasons. It is also allowed to hot-reload empty groups by adding or removing rules. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-25 14:46:51 +03:00
Aliaksandr Valialkin	2d7d67372f	docs/Cluster-VictoriaMetrics.md: add links with the explanation of active time series and series churn rate	2021-10-24 18:39:16 +03:00
Aliaksandr Valialkin	30ade321a6	docs/CaseStudies.md: fix a link to AbisoGaming case study	2021-10-24 18:37:19 +03:00
Aliaksandr Valialkin	d8ce86d685	app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738	2021-10-24 13:32:25 +03:00
Aliaksandr Valialkin	7ec253a3da	docs/CHANGELOG.md: typo fix	2021-10-22 21:13:03 +03:00
Aliaksandr Valialkin	9e0a67cf13	go.mod: remove outdated replacement	2021-10-22 19:47:10 +03:00
Aliaksandr Valialkin	1bdfa475f3	docs/CHANGELOG.md: cut v1.68.0	2021-10-22 19:38:01 +03:00
Aliaksandr Valialkin	e18aa6e07e	deployment/docker: update Grafana from v8.2.0 to v8.2.2	2021-10-22 19:30:06 +03:00
Aliaksandr Valialkin	064abfba8c	vendor: `make vendor-update`	2021-10-22 19:28:27 +03:00
Aliaksandr Valialkin	1780c25e05	docs/vmauth.md: `make docs-sync`	2021-10-22 19:28:27 +03:00
Yurii Kravets	f7cb850f8d	Updated docker-compose.yml (#1730 ) * Updated docker-compose.yml Added one more Vmstorage to compose and pointed vmselect and vminsert to it. * Update prometheus.yml According to requested changes for new vmstorage * Update docker-compose Added port mapping * Updated docker-compose Set same ports for both storages * Update prometheus.yml Changed target for vmstorage1 * Updated docker compose Changed the flags for vmstorage1 * Update docker compose Changed flags for vmisert and vmselect to point to vmstorage1 * Update docker compose Made the vmstorage names uniform * Update prometheus.ymk * Update docker-compose.yml Removed additional flags as they are using default values. * Update prometheus.yml Put vmstorage targets under the same job * Update prometheus.yml Changed targets format * Update docker-compose.yml Set two different volumes for every vmstorage node * Update deployment/docker/prometheus.yml * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 19:26:02 +03:00
Aliaksandr Valialkin	559dd996c4	app/vmauth: add ability to specify http headers to send in requests to backends See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1736	2021-10-22 19:11:35 +03:00
Aliaksandr Valialkin	b08f51f5d3	lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set The response body isn't used if -promscrape.noStaleMarkers is set after the commit `2876137c92` , so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694	2021-10-22 16:49:21 +03:00
Aliaksandr Valialkin	fcdb0c415e	docs/CHANGELOG.md: document `43a7984cd8`	2021-10-22 14:00:50 +03:00
Aliaksandr Valialkin	6bc10f0623	lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets This is a follow-up after `9882cda8b9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728	2021-10-22 13:55:39 +03:00
Aliaksandr Valialkin	a8bcc3c276	lib/promscrape: reduce memory usage if `-promscrape.noStaleMarkers` command-line flag is passed Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled. This should reduce memory usage when the scraped targets return large responses.	2021-10-22 13:22:08 +03:00
Roman Khavronenko	e0f21d6000	vmalert: correctly calculate alert ID including extra labels (#1734 ) Previously, ID for alert entity was generated without alertname or groupname. This led to collision, when multiple alerting rules within the same group producing same labelsets. E.g. expr: `sum(metric1) by (job) > 0` and expr: `sum(metric2) by (job) > 0` could result into same labelset `job: "job"`. The issue affects only UI and Web API parts of vmalert, because alert ID is used only for displaying and finding active alerts. It does not affect state restore procedure, since this label was added right before pushing to remote storage. The change now adds all extra labels right after receiving response from the datasource. And removes adding extra labels before pushing to remote storage. Additionally, change introduces a new flag `Restored` which will be displayed in UI for alerts which have been restored from remote storage on restart.	2021-10-22 12:31:35 +03:00
Aliaksandr Valialkin	ad40e70d39	docs/CHANGELOG.md: document `a3684fe3de`	2021-10-22 12:29:27 +03:00
Nikolay	83e1dfccba	adds tab as second separator for graphite text protocol (#1733 ) * adds tab as second separator for graphite text protocol * changes indexFunc for indexAny * Update lib/protoparser/graphite/parser_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:29:27 +03:00
Yury Molodov	3ccc37a98b	vmui: query history (#1732 ) * feat: add query history * fix: change detect keyUp for nav query history * feat: set default query history * app/vmselect/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:29:27 +03:00
Aliaksandr Valialkin	d56e676d71	lib/flagutil: do not expose sensitive info (passwords, keys and urls) at /flags page	2021-10-20 00:51:15 +03:00
Aliaksandr Valialkin	5705f4b6d1	lib/httpserver: expose command-line flags at `/flags` page This should simplify debugging. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-20 00:46:54 +03:00
Aliaksandr Valialkin	a105b71116	lib/envflag: use flag.Set for setting the flags from env vars This should make visible the set flags at flag.Visit(), which is used later for logging and exporting the `is_set` label for these flags at /metrics page	2021-10-20 00:46:53 +03:00
Aliaksandr Valialkin	93511b4be7	lib/storage: log a warning when the -storageDataPath has less than -storage.minFreeDiskSpaceBytes This should improve the debuggability of the readonly feature. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727	2021-10-19 23:58:09 +03:00
Roman Khavronenko	ca49853664	vmalert: make group.ID() thread-safe (#1726 ) Commit fixes potential race condition when group update and generating of ID() happens simultaneously. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-19 16:45:52 +03:00
Roman Khavronenko	627224d493	vmalert: properly init SIGHUP listener before starting group manager (#1725 ) Regression was introduced during code refactoring. It potentially could lead to situation when SIGHUP signals were ignored while vmalert was still busy with initing group manager. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-19 16:36:08 +03:00
Aliaksandr Valialkin	0a1982b294	app/vmauth: fix metric name prefixes: `vmagent` -> `vmauth`	2021-10-19 15:30:16 +03:00
Aliaksandr Valialkin	5df7ef323f	docs/Single-server-VictoriaMetrics.md: add a link to VMUI at VictoriaMetrics playground	2021-10-19 14:42:35 +03:00
Aliaksandr Valialkin	ea69eef375	lib/promscrape/discovery/kubernetes: log a warning if `role: endpoints` discovers more than 1000 targets per a single endpoint In this case `role: endpointslice` must be used instead. See the following references: * https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity * https://github.com/kubernetes/kubernetes/pull/99975 * https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398	2021-10-19 13:22:28 +03:00
Aliaksandr Valialkin	948e9b8f50	docs/CHANGELOG.md: document `146a5b504c`	2021-10-19 11:24:44 +03:00
Aliaksandr Valialkin	a7de3712e6	docs/CHANGELOG.md: document `cbcc622786`	2021-10-19 09:00:05 +03:00
Aliaksandr Valialkin	e4ebcebc8a	deployment/docker/alerts.yml: formatting fixes after `865a60f13e`	2021-10-19 09:00:05 +03:00
Nikolay	e84a063209	changes job source for /target api (#1723 ) use jobNameOriginal instead of relabeled as prometheus does https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707	2021-10-19 09:00:05 +03:00
Roman Khavronenko	d763837130	dashboards: add cardnilaity limiter panels for vmagent (#1720 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-19 09:00:05 +03:00
Yurii Kravets	34f52de3a5	Update alerts.yml Added Series Limit day\hour alerts	2021-10-19 09:00:05 +03:00
Aliaksandr Valialkin	79709e2586	vendor: return back the previous google.golang.org/genproto version, since the latest version leads to compile errors The following errors: vendor/cloud.google.com/go/storage/storage.go:1447:53: o.GetCustomerEncryption().GetKeySha256 undefined (type "google.golang.org/genproto/googleapis/storage/v2".Object_CustomerEncryption has no field or method GetKeySha256) vendor/cloud.google.com/go/storage/writer.go:439:10: q.GetCommittedSize undefined (type "google.golang.org/genproto/googleapis/storage/v2".QueryWriteStatusResponse has no field or method GetCommittedSize)	2021-10-18 15:37:29 +03:00
Aliaksandr Valialkin	c92746bb01	vendor: `make vendor-update`	2021-10-18 15:23:46 +03:00
Yury Molodov	744f6a98eb	vmui: features (#1711 ) * feat: initial uPlot graph * feat: add zoom/pan for graph * fix: add zoom by ctrl/mac * fix: remove unused code * feat: add toggle cache for fetch * feat: add fix y-axis limits * fix: stop point events while panning * fix: change getting cursor position when scaling * feat: add cursor tooltip to graph * fix: uninstall chart.js * fix: change link for create an issue * fix: set default cache value to true * app/vmalert: follow-up after `0e2486df56` * docs/CHANGELOG.md: document `5416e18007` * app/vmui: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-18 15:19:01 +03:00
Roman Khavronenko	44a88b7458	vmalert: remove extra `/` from path in WEB interface (#1717 ) The extra `/` may cause issues when additional path prefixes are configured. Also, removing it makes it consistent with the rest of declarations. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-18 15:19:01 +03:00
Roman Khavronenko	93d4ef1239	vmctl: follow-up after `95d1d38595` (#1718 ) Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-18 15:19:01 +03:00
Aliaksandr Valialkin	f87d43a5e4	docs/CHANGELOG.md: document `5416e18007`	2021-10-18 15:08:44 +03:00
Aliaksandr Valialkin	9273b83cd0	app/vmalert: follow-up after `0e2486df56`	2021-10-18 15:08:42 +03:00
Miro Prasil	c0853d4bf8	vmctl influx convert bool to number (#1714 ) vmctl: properly convert influx bools into integer representation When using vmctl influx, the import would fail importing boolean fields with: ``` failed to convert value "some".0 to float64: unexpected value type true ``` This converts `true` to `1` and `false` to `0`. Fixes #1709	2021-10-18 14:59:17 +03:00
Alexander Rickardsson	0e1dbcd039	vmalert: add disablePathAppend to remote read (#1712 ) * vmalert: add disablePathAppend to remoteRead * docs: add docs for remoteRead.disablePathAppend	2021-10-18 14:59:17 +03:00
Alexander Rickardsson	63571e1334	vmalert: Redact passwords from error messages (#1713 )	2021-10-18 14:59:17 +03:00
Aliaksandr Valialkin	ed994873fd	app/vmselect/promql: randomize the static selection of time series returned from `limitk()` Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series. Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.	2021-10-16 21:16:34 +03:00
Aliaksandr Valialkin	fbcc8b5c7d	lib/promscrape: set `honor_timestamps: true` by default if this option isnt set explicitly in scrape configs This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-10-16 20:48:53 +03:00
Aliaksandr Valialkin	102dd91ddd	docs/CaseStudies.md: add Smarkets case study	2021-10-16 20:00:32 +03:00
Aliaksandr Valialkin	6e1d7d3c9c	docs/CaseStudies.md: add Fly.io case study	2021-10-16 20:00:32 +03:00
Aliaksandr Valialkin	043c7119c8	docs/CaseStudies.md: add a case study for Razorpay	2021-10-16 20:00:32 +03:00
Aliaksandr Valialkin	3e12eeaa3b	docs/CaseStudies.md: add AbiosGaming	2021-10-16 20:00:32 +03:00
Aliaksandr Valialkin	24ffb606be	docs/CaseStudies.md: add Percona case study	2021-10-16 19:14:15 +03:00
Aliaksandr Valialkin	0a9be5ef9d	lib/promscrape: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics per each scrape target with the enabled unique series limiter	2021-10-16 19:14:13 +03:00
Aliaksandr Valialkin	f1803573c3	vendor: update github.com/valyala/gozstd from v1.13.0 to v1.14.1 This should reduce memory usage in vmagent when compressing large scrape responses in stream parsing mode	2021-10-16 19:14:07 +03:00
Aliaksandr Valialkin	99011c6b63	lib/promscrape: always initialize http client for stream parsing mode Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:19:48 +03:00
Aliaksandr Valialkin	0f4fda1bda	lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse This should reduce memory usage when scraping targets with big response bodies.	2021-10-16 13:00:11 +03:00
Aliaksandr Valialkin	da0adf446d	app/vmselect/promql: typo fix in comment: didsn't -> didn't	2021-10-16 13:00:10 +03:00
Aliaksandr Valialkin	0452a8d4e8	lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode at the cost of the increased memory usage, since now the potentially big response is kept in the lastScrape byte slice per each scrapeWork. In practice the memory usage increase shouldn't be big, since the response size is usually much smaller than the parsed metrics from this response after the relabeling, which usually adds a big pile of target-specific labels per each metric.	2021-10-15 15:26:24 +03:00
Aliaksandr Valialkin	3e9beb0f8d	lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name This is a follow-up for `31b42b30b6`	2021-10-15 12:27:31 +03:00
Aliaksandr Valialkin	72c79d7689	docs/FAQ.md: improve wording on why MetricsQL isnt 100% compatible with PromQL	2021-10-14 16:23:17 +03:00
Aliaksandr Valialkin	ae1ee5ba4a	docs/CHANGELOG.md: document the change at `7fcbd3fa4b`	2021-10-14 14:38:14 +03:00
Aliaksandr Valialkin	1beb38d476	docs/FAQ.md: add an entry explaining why MetricsQL isn't 100% compatible with PromQL	2021-10-14 12:50:24 +03:00
Aliaksandr Valialkin	25421fa2ae	lib/promscrape: add `-promscrape.minResponseSizeForStreamParse` command-line option for automatic switching to stream parsing mode when scraping targets with big responses This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes. This is common case in Kubernetes monitoring.	2021-10-14 12:30:55 +03:00
Aliaksandr Valialkin	bee130cc78	lib/promscrape: return error if `sample_limit` or `series_limit` options are set when stream parsing mode is enabled	2021-10-14 12:30:54 +03:00
Roman Khavronenko	f393145843	Adjust `http.Transport.MaxIdleConns` setting for vmauth/vmalert services (#1704 ) * vmalert: adjust `http.Transport.MaxIdleConns` value accordingly to `http.Transport.MaxIdleConnsPerHost` `http.Transport.MaxIdleConnsPerHost` setting is controlled by `datasource.maxIdleConnections` flag, while `http.Transport.MaxIdleConns` is inherited from DefaultTransport and is equal to `100`. The fix adjusts `http.Transport.MaxIdleConns` value if it is lower than `http.Transport.MaxIdleConnsPerHost`. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmauth: adjust `http.Transport.MaxIdleConns` value accordingly to `http.Transport.MaxIdleConnsPerHost` `http.Transport.MaxIdleConnsPerHost` setting is controlled by `maxIdleConnsPerBackend` flag, while `http.Transport.MaxIdleConns` is inherited from DefaultTransport and is equal to `100`. The fix adjusts `http.Transport.MaxIdleConns` value if it is lower than `http.Transport.MaxIdleConnsPerHost`. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-10-13 19:23:52 +03:00
Aliaksandr Valialkin	73dbad3c2a	docs/CHANGELOG.md: document `e3c8304deb`	2021-10-13 16:44:34 +03:00
Aliaksandr Valialkin	5b7d90d178	lib/promscrape: add ability to show the original labels for discovered targets at /targets page See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698	2021-10-13 16:44:34 +03:00
Roman Khavronenko	14995eece6	vmalert: add Source link to alerts UI (#1701 ) The source link is controlled by `external.url` and `external.alert.source` flags, in the same way as for alertmanager notifications. The source link is added to Alerts list view, and specific Alert view.	2021-10-13 15:26:57 +03:00
Aliaksandr Valialkin	9568961054	docs/MetricsQL.md: add missing blank line before the link to github.com/VictoriaMetrics/metricsql package	2021-10-13 15:11:46 +03:00
Roman Khavronenko	5dab25e8ad	lib/promscrape: make errcheck happy (#1703 )	2021-10-13 15:11:45 +03:00
Aliaksandr Valialkin	c3a729d458	lib/promscrape: shard targets among cluster nodes after relabeling is applied This guarantees that targets with the same set of labels go to the same vmagent node. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495	2021-10-12 17:06:37 +03:00
Aliaksandr Valialkin	aeedfe2fe2	app/vmagent: expose -promscrape.config contents at /config page as Prometheus does See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-12 16:27:37 +03:00
Aliaksandr Valialkin	1e56d19c09	docs/FAQ.md: add a solution against high churn rate	2021-10-12 10:33:08 +03:00
Roman Khavronenko	49a7ff62df	docs: mention "PromQL compliance" in MetricsQL docs (#1691 )	2021-10-11 21:52:28 +03:00
Aliaksandr Valialkin	4878fcedac	vendor: `make vendor-update`	2021-10-11 21:52:27 +03:00
Aliaksandr Valialkin	48c3668b3d	app/vmselect/promql: add `atan2` binary operator, which is going to be added in Prometheus 2.31 See https://github.com/prometheus/prometheus/pull/9248	2021-10-11 21:17:23 +03:00
Aliaksandr Valialkin	b3d28f3872	app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31 See https://github.com/prometheus/prometheus/issues/9233	2021-10-11 21:17:21 +03:00
Aliaksandr Valialkin	d00f8b8800	docs/MetricsQL.md: clarify docs for union() function	2021-10-11 17:41:10 +03:00
Aliaksandr Valialkin	84aa08d93a	lib/promscrape: use Prometheus format for target labels at `/targets` page This should simplify copy-pasting the labels to/from PromQL / MetricsQL	2021-10-11 12:42:18 +03:00
Denys Holius	6199bd28a2	Added some fixes (#1690 ) * removed not needed description * added some fixes and fixed typos	2021-10-11 12:42:18 +03:00
Aliaksandr Valialkin	f17c4b74c5	docs/MetricsQL.md: add a link to https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e	2021-10-11 11:02:03 +03:00
Roman Khavronenko	de4fbe9402	docs: add "PromQL compliance" article (#1689 ) * docs: add "PromQL compliance" article * Update docs/Articles.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-11 11:01:59 +03:00
Aliaksandr Valialkin	ab3dcf3f77	deployment/docker/docker-compose.yml: upgrade Grafana from v8.1.2 to v8.2.0	2021-10-08 20:37:32 +03:00
Aliaksandr Valialkin	f1e1d20ac4	app/vmselect/promql: consistently return the same set of time series from `limitk()` function This is the expected behaviour by most users.	2021-10-08 19:55:29 +03:00
Aliaksandr Valialkin	a7a1305395	lib/storage: fix unaligned access on 32-bit architectures. The bug has been introduced at `a171916ef5`	2021-10-08 19:38:20 +03:00
Denys Holius	d1ecffb0f6	Added fixes and improvements (#1677 ) * added guide for VM operator * Update docs/guides/getting-started-with-vm-operator.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/getting-started-with-vm-operator.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Fixed different typos and added improvements from proposals * move remoteWrite.url to other place * fixed typo * rephrased vminsert explanation * remove not needed parameters for default setup Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2021-10-08 18:58:42 +03:00
Aliaksandr Valialkin	39aa1217c6	deployment/docker: update Go builder version from Go1.17.1 to Go1.17.2 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.2+label%3ACherryPickApproved	2021-10-08 17:53:06 +03:00
Aliaksandr Valialkin	20fa4b01cb	vendor: `make vendor-update`	2021-10-08 16:06:31 +03:00
Aliaksandr Valialkin	aae6f7dfd0	docs/CHANGELOG.md: cut v1.67.0	2021-10-08 16:00:43 +03:00
Aliaksandr Valialkin	a47754b689	lib/protoparser/clusternative: typo fix after `4fddcf4c83`	2021-10-08 15:38:47 +03:00
Aliaksandr Valialkin	cf87050d83	app/vmui: follow-up after `7bfb44113e` * Run `vmui-update` * Document the changes in README.md and CHANGELOG.md	2021-10-08 15:11:09 +03:00
Yury Molodov	41a83a1cc6	vmui: use uPlot as default engine for graph (#1683 ) * feat: initial uPlot graph * feat: add zoom/pan for graph * fix: add zoom by ctrl/mac * fix: remove unused code	2021-10-08 15:11:07 +03:00
Aliaksandr Valialkin	4fddcf4c83	app/{vminsert,vmstorage}: follow-up after `a171916ef5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-10-08 14:09:51 +03:00
Nikolay	a171916ef5	Adds read-only mode for vmstorage node (#1680 ) * adds read-only mode for vmstorage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269 * changes order a bit * moves isFreeDiskLimitReached var to storage struct renames functions to be consistent change protoparser api - with optional storage limit check for given openned storage * renames freeSpaceLimit to ReadOnly	2021-10-08 12:52:56 +03:00
Aliaksandr Valialkin	cbf01f7384	app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow The list of functions, which can adjust lookbehind window is more limited than the rest of functions, so it is better from maintainability and readability PoV using the allowlist instead of blocklist.	2021-10-07 13:19:22 +03:00
Aliaksandr Valialkin	7e6ec0bbe9	app/vmselect/promql: return back the behaviour for `deriv()` function when the lookbehind window doesnt contain enough points It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d` contains less than 2 samples in the same way as `rate()` does. This is a follow-up after `3e084be06b` .	2021-10-07 12:55:58 +03:00
Roman Khavronenko	0a8804d6aa	app/vmselect: make `predict_linear` and `deriv` compatible with Prometheus (#1681 ) Previously, `predict_linear` returned slightly different results comparing to Prometheus. The change makes linear regression algorithm compatible with Prometheus. `deriv` was excluded from the list of functions which can adjust the time window for the same reasons.	2021-10-07 12:55:56 +03:00
Aliaksandr Valialkin	64b6f3f1c8	app/vminsert: fix uneven distribution of time series among storage nodes Use distinct seed for distribution hash calculations on the second level of vminsert nodes. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672	2021-10-07 12:22:39 +03:00
Aliaksandr Valialkin	cea79d2013	docs/guides: follow-up after `05a1396247`	2021-10-06 14:47:28 +03:00
Thomas Danielsson	8d26b0f3f5	fix: typo metric_relabel_configs (#1674 ) metric_ralabel_configs -> metric_relabel_configs	2021-10-06 14:47:26 +03:00
Ziqi Zhao	1db3aeab36	fix some typos (#1678 ) Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>	2021-10-06 14:43:56 +03:00
Aliaksandr Valialkin	bb45d747ad	vendor: `make vendor-update`	2021-10-05 10:31:30 +03:00
Aliaksandr Valialkin	89700433cc	docs/vmagent.md: update docs after 3e9a939a990c8b608414388c96f68eb062364ae7	2021-10-05 10:23:37 +03:00
Aliaksandr Valialkin	f434fcbbd7	vendor: `make vendor-update`	2021-09-30 17:54:45 +03:00
Aliaksandr Valialkin	a9420317d9	docs/vmagent.md: document how to write data to Kafka	2021-09-30 17:46:23 +03:00
Aliaksandr Valialkin	522a404b79	lib/promscrape: reduce memory allocations in mergeLabels() after `48e3e6c8df`	2021-09-30 16:56:43 +03:00
Aliaksandr Valialkin	ec6eb03d65	app/vmselect/promql: add `topk_last` and `bottomk_last` functions	2021-09-30 13:23:27 +03:00
Roman Khavronenko	95ddfda894	app/vmselect: fix binary comparison func (#1667 ) The fix makes the binary comparison func to check for NaNs before executing the actual comparison. This prevents VM to return values for non-existing samples for expressions which contain bool comparisons. Please see added test for example.	2021-09-30 13:04:10 +03:00
Roman Khavronenko	57eb5ef194	app/vmselect: fix `testRowsEqual` func NaN checks (#1666 ) It appeared, that `testRowsEqual` NaN comparison was incorrect. The fix caused some tests to fail. Please see the change and tests updated. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-30 13:04:08 +03:00
Aliaksandr Valialkin	7b69d478ec	lib/protoparser: go fmt	2021-09-29 21:17:49 +03:00
Aliaksandr Valialkin	d51118e226	app/vminsert: document that -relabelConfig is reloaded on SIGHUP signal	2021-09-29 21:17:25 +03:00
Aliaksandr Valialkin	6167890d0e	lib/protoparser/prometheus: compare invalid Prometheus lines in full	2021-09-29 19:41:23 +03:00
Aliaksandr Valialkin	ae76ceb5f1	docs/CHANGELOG.md: link to Kafka integration docs	2021-09-29 12:31:59 +03:00
Aliaksandr Valialkin	8dcf814c48	app/{vmbackup,vmrestore}: switch from `gcs://...` to `gs://...` urls for backups to GCS The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls, while leaving support for `gcs://` urls for backwards compatibility.	2021-09-29 12:12:37 +03:00
Aliaksandr Valialkin	f35f9188bc	docs/vmagent.md: update Telegraf config in the section about Kafka	2021-09-29 12:01:46 +03:00
Aliaksandr Valialkin	16d7ee7b44	docs/vmagent.md: add docs about reading metrics from Kafka	2021-09-29 01:46:40 +03:00
Yury Molodov	a6037a8ce8	vmui: fixed bug with time range (time zone) (#1661 ) * fix: set date in query string in utc format * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 01:01:27 +03:00
Nikolay	6ca31acb76	changes vmagent api (#1656 ) * changes vmagent api adds auth.Token to promremotewrite InsertHandlerReader changes remoteWrite client constructor, allows to use multiple remoteWriteUrl schemes, like kafka:// changes url path concatenation for tenant remoteWrite Update app/vmagent/remotewrite/client.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * Update app/vmagent/remotewrite/remotewrite.go * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 00:55:14 +03:00
Roman Khavronenko	444a8882f4	app/vmselect: disable time-window adjustment for `min/max_over_time` funcs (#1658 ) Adjustment results into discrepancy between Prometheus and VM on time windows smaller than scrape interval. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:43:54 +03:00
Aliaksandr Valialkin	428c03c3b8	docs/CHANGELOG.md: document the bugfix from `de810031bf`	2021-09-29 00:42:21 +03:00
Roman Khavronenko	f46f847531	app/vmselect: always return zero for `stddev` func if there is only one value (#1659 ) The fix will always return zero if received set of items consists of one element only, which also means no deviation. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:42:19 +03:00
Roman Khavronenko	72468c9d75	app/vmselect: return NaN instead of 0 for empty value sets (#1660 ) The change affects `count/stddev/stdvar_over_time` funcs and makes them to return NaN instead of zero when there is no datapoints in a time window. This is needed for improving compatibility with Prometheus. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:38:24 +03:00
Roman Khavronenko	c8ac8f44b2	app/vmselect: rm `quantile_over_time` fast-path optimisations (#1662 ) The removed fast path optimisations weren't consistent with `quantile` function behavior and results into discrepancy. Specifically, results didn't match in cases when: * 0 < phi > 1; * values contain only one element. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-29 00:35:59 +03:00
Aliaksandr Valialkin	2577e7c306	docs/CHANGELOG.md: document `3d17112a7e`	2021-09-29 00:33:41 +03:00
Nikolay	9be5689b3f	changes auth validation for openstack (#1663 ) * changes auth validation for openstack must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 00:33:38 +03:00
Aliaksandr Valialkin	4e65bfcc00	app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206	2021-09-29 00:12:26 +03:00
Yury Molodov	80df31b2ee	vmui: use Chart.js as default engine for graph (#1634 ) * feat: add Plotly as default engine for graph * fix: remove unused components * feat: use Chart.js as default engine graph * fix: correct styles for loader * feat: add zoom/pan for chart * feat: add height for chart * fix: remove unused code * fix: remove empty units from duration * fix: change debounce for pan to 500ms * fix: add utility for plugins register globally * fix: optimize render graph * feat: add buffer data for zoom * fix: add limits for zoom in/out * fix: change update data while zooming * app/vmselect: `make vmui-update` Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-27 22:27:49 +03:00
Aliaksandr Valialkin	4b3951fd86	app/vmselect/promql: add `rollup_scrape_interval(m[d])` function It calculates the min, max and avg scrape intervals for m over the given lookbehind window d	2021-09-27 19:21:55 +03:00
Aliaksandr Valialkin	eff31c10ec	app/vmselect/promql: follow-up after `526dd93b32` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612	2021-09-27 18:57:03 +03:00
Roman Khavronenko	6061464d80	app/vmselect: `quantile` func compatiblity with Prometheus (#1646 ) * app/vmselect: `quantile` func compatiblity with Prometheus The `quantile` func was previously calculated by https://github.com/valyala/histogram package. The result of such calculation was always the closest real value to requested quantile. While in Prometheus implementation interpolation is used. Such difference may result into discrepancy in output between Prometheus and VictoriaMetrics. This commit adds a Prometheus-like `quantile` function. It also used by other functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625 Signed-off-by: hagen1778 <roman@victoriametrics.com> * app/vmselect: `quantile` review fixes * quantile functions were split into multiple to provide different API for already sorted data; * float64sPool is used for reducing allocations. Items in pool may have different sizes, but defining a new pool was complicates due to name collisions; Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-27 18:57:02 +03:00
Aliaksandr Valialkin	469820873f	vendor: `make vendor-update`	2021-09-27 18:01:46 +03:00
Aliaksandr Valialkin	a957d420f4	docs/BestPractices.md docs/BestPractices.md: update the doc	2021-09-27 17:51:57 +03:00
Aliaksandr Valialkin	a746fe8778	app/vmselect/promql: follow-up after `57b3320478`	2021-09-24 01:25:04 +03:00
Roman Khavronenko	7ac0559d4f	app/vmselect: make sorting for query result similar to Prometheus (#1647 ) * app/vmselect: make sorting for query result similar to Prometheus Updated sorting allows to get the order of series in result similar or equal to what Prometheus returns. The change is needed for compatibility reasons. * Update app/vmselect/promql/exec_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-24 01:25:03 +03:00
Aliaksandr Valialkin	1115c2f235	app/vmselect/promql: align the behavior of `or`, `and` and `unless` operators with `on (labels)` modifier to Prometheus Previously VictoriaMetrics could return unexpected result of the right-hand side operand had multiple time series with the given set of labels mentioned in `on(labels)`. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643	2021-09-24 00:56:35 +03:00
Nikolay	8148bfc06d	moves prod images build into alpine container with musl (#1640 ) adds gcc and musl-dev to builder container	2021-09-24 00:20:33 +03:00
Aliaksandr Valialkin	6fbfa026ca	docs/CHANGELOG.md: cut v1.66.2	2021-09-23 22:53:27 +03:00
Aliaksandr Valialkin	d15d036a5a	lib/storage: properly handle `{__name__=~"prefix(suffix1\|suffix2)",other_label="..."}` queries They were broken in the commit `00cbb099b6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644	2021-09-23 21:52:31 +03:00
Aliaksandr Valialkin	f93146c81e	docs: `make docs-sync`	2021-09-23 20:55:29 +03:00
Aliaksandr Valialkin	7edb65a9a3	docs/CHANGELOG.md: document `0e35fc9538` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1641	2021-09-23 20:47:56 +03:00
Roman Khavronenko	f65ca2ac05	app/vmctl: fix misleading comment about cluster version for native mode (#1648 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1637 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-23 20:47:56 +03:00
Roman Khavronenko	1bdccfc2bf	app/vmalert: remove unnecessary `omitempty` tag for `interval` param (#1649 ) `omitempty` tag resulted into skipping this param on marshaling, which was used as a checksum for groups configuration. Since on config reload checksums are compared before applying changes, any change to `interval` only didn't trigger config reload. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1641 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2021-09-23 20:47:56 +03:00
Aaron France	e398d83829	fix: typo in vmagent.md (#1642 )	2021-09-23 20:47:56 +03:00
Aliaksandr Valialkin	3b501ca859	vendor: `make vendor-update`	2021-09-23 15:06:57 +03:00
Aliaksandr Valialkin	d8de26bbfd	lib/promscrape: add `vm_promscrape_max_scrape_size_exceeded_errors_total` metric for counting of the failed scrapes due to the exceeded response size Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin	9b76515216	docs/CaseStudies.md: add a case study for Grammarly	2021-09-23 13:12:20 +03:00
Aliaksandr Valialkin	e128915c6e	docs/Articles.md: add https://cer6erus.medium.com/superset-bi-with-victoria-metrics-a109d3e91bc6	2021-09-23 10:44:14 +03:00
Aliaksandr Valialkin	baeb2fce3a	vendor: update github.com/valyala/gozstd from v1.12.0 to v1.13.0	2021-09-22 20:06:52 +03:00
Aliaksandr Valialkin	5e5d5fa8fb	docs/CaseStudies.md: fix a link to third-party articles	2021-09-22 03:53:08 +03:00
Aliaksandr Valialkin	47422e9918	docs/CHANGELOG.md: cut v1.66.1	2021-09-22 01:47:26 +03:00
Aliaksandr Valialkin	25361d176d	docs/CHANGELOG.md: document `9ca1cbced1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635	2021-09-21 23:17:26 +03:00
Aliaksandr Valialkin	86bafe796c	lib/httpserver: add `-enterprise` and/or `-cluster` suffixes to `short_version` label of `vm_app_version` metric See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635	2021-09-21 23:12:50 +03:00
Aliaksandr Valialkin	c3e1f87048	lib/promrelabel: fix parsing `regex: true` in relabeling rules	2021-09-21 23:01:40 +03:00
Roman Khavronenko	aa052097cf	app/vmalert: support `http.pathPrefix` flag in UI (#1636 ) The change makes UI to respect `http.pathPrefix` flag for API or navigation items links.	2021-09-21 18:15:08 +03:00
Aliaksandr Valialkin	0e7b2b724f	vendor: temporarily stick to v0.93.3 for cloud.google.com/go until the binary size bloat issue is resolved This returns back VictoriaMetrics binary size from 24Mb to 18Mb. See https://github.com/googleapis/google-cloud-go/issues/4783	2021-09-21 18:13:33 +03:00
Aliaksandr Valialkin	3f1104ea74	app/vmselect: fix accessing `/graphite/*` endpoints	2021-09-21 14:06:18 +03:00
Aliaksandr Valialkin	e8700b57b9	docs/vmagent.md: typo fix	2021-09-20 16:49:32 +03:00
Aliaksandr Valialkin	233c6a8a82	docs/vmagent.md: typo fixes in `Prometheus staleness markers` docs	2021-09-20 16:44:42 +03:00
Aliaksandr Valialkin	d43ebb8314	docs/CHANGELOG.md: cut v1.66.0	2021-09-20 15:20:36 +03:00
Nikolay	dd53abf36d	changes protoparser apis for accepting reading from io.Reader (#1624 ) adds InsertHandlerForReader apis to vmagent	2021-09-20 14:54:20 +03:00
Aliaksandr Valialkin	3b9fd99735	vendor: `make vendor-update`	2021-09-20 14:44:49 +03:00
n4mine	a5fcc9e356	fix: typo, dddresses -> addresses (#1630 )	2021-09-20 14:27:34 +03:00
Aliaksandr Valialkin	4150772ef8	docs/Articles.md: add a link to `Open-source strategy at VictoriaMetrics`	2021-09-20 11:46:38 +03:00
Aliaksandr Valialkin	76c650bbda	app/vmselect/promql: add `quantiles_over_time("phiLabel", phi1, ..., phiN, m[d])` function for calculating multiple quantiles at once	2021-09-17 23:35:04 +03:00
Aliaksandr Valialkin	727521d20a	docs/CHANGELOG.md: document `0e09fdb8b0`	2021-09-17 18:47:39 +03:00
Nikolay	1ab2f844a2	makes filters optional for ec2 api requests (#1627 ) filters can be applied only for DescribeInstances requests, like prometheus does. related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin	81d24045c2	app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function This function calculates multiple quantiles over the given buckets at once Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-09-17 13:39:06 +03:00
Aliaksandr Valialkin	9333c4111e	docs/CHANGELOG.md: document the change in enterprise apps, which allows passing `-version` without `-eula` flag See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621	2021-09-17 12:38:22 +03:00
Aliaksandr Valialkin	4ed7548ced	app/vmselect/promql: optimize quantiles() calculation Calculate quantiles in one go instead of calculating each quantile individually	2021-09-17 12:38:21 +03:00
Aliaksandr Valialkin	f9910113b5	docs/FAQ.md: extend `VictoriaMetrics vs TimescaleDB` section with real user experience See also https://github.com/timescale/promscale/issues/427 , which is mentioned in the https://abiosgaming.com/press/high-cardinality-aggregations/	2021-09-16 20:47:15 +03:00
Roman Khavronenko	a07fa92ef2	vmalert: add new metric `vmalert_remotewrite_flush_duration_seconds` (#1622 )	2021-09-16 14:01:24 +03:00
Roman Khavronenko	cec67a3129	docs: fix indentation for FAQ document (#1620 )	2021-09-16 14:01:22 +03:00
f41gh7	348f705d95	follow-up after e2f99214f3e518f79c1c96ff918820a69d350ba1	2021-09-16 13:56:28 +03:00
Roman Khavronenko	05bf6cca82	vmselect: add stub for `graphite/functions` requests The request `graphite/functions` is sent by Grafana datasource and results into error logs in vmselect.	2021-09-16 13:51:37 +03:00
Aliaksandr Valialkin	3f8de4f82f	app/vmselect/promql: add `mad(q)` and `outliers_mad(tolerance, q)` functions to MetricsQL	2021-09-16 13:35:38 +03:00
Aliaksandr Valialkin	a4eaf1a0a1	app/vmselect/prometheus: make more clear log messages for errors during sending data to remote clients	2021-09-16 13:35:37 +03:00
Aliaksandr Valialkin	7d910c6a51	vendor: `make vendor-update`	2021-09-15 18:24:08 +03:00
Aliaksandr Valialkin	7ad54041fe	app/{vminsert,vmselect}: automatically add missing port in `-storageNode` lists passed to `vminsert` and `vmselect` This should simplify manual setup of the cluster according to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup	2021-09-15 18:04:30 +03:00
Aliaksandr Valialkin	9ebccddfc2	docs/CHANGELOG.md: document `777ff75874`	2021-09-15 17:45:18 +03:00
Aliaksandr Valialkin	794d941125	app/vmui: follow-up after `777ff75874` The commit contains the following changes: - Show vmui when requesting /graph page in order to be compatible with Prometheus datasource in Grafana. - Properly encode query args at vmui url. - Set the number of points on the graph to the number of horizontal pixels divided by 2. Previously it was hardcoded to 30. - Do not save server url to persistent storage at browser, since it should be always obtained from the url. - Run `make vmui-update` for updating vmui embedded into VictoriaMetrics.	2021-09-15 17:41:00 +03:00
Yury Molodov	df87362ef6	vmui: change query params compatible with prometheus (#1619 ) * feat: change url params for compatible prometheus * style: add comment for TimeParams * fix: change get default server for single version * fix: change function for get query string value	2021-09-15 16:36:39 +03:00
Aliaksandr Valialkin	76109eb0e6	vendor: update github.com/valyala/quicktemplate from v1.6.3. to v1.7.0	2021-09-15 09:35:40 +03:00
Aliaksandr Valialkin	de4f2c9263	vendor: update github.com/VictoriaMetrics/fastcache from v1.6.0 to v1.7.0	2021-09-15 09:35:40 +03:00
Aliaksandr Valialkin	91fd06a815	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.16 to v1.1.0	2021-09-15 09:35:40 +03:00
Aliaksandr Valialkin	e67b4fca11	vendor: update github.com/valyala/histogram from v1.1.2 to v1.2.0 This fixes the non-repeatable quantile_over_time() results when the number of input samples exceeds 1000. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612	2021-09-15 09:35:40 +03:00
Roman Khavronenko	286b79dc48	vmalert: create basic auth config only if args aren't empty (#1618 ) * vmalert: create basic auth config only if args aren't empty follow-up after `68721f6` * vmalert: make lint happy	2021-09-15 09:35:38 +03:00
Roman Khavronenko	45cce85bf4	docs: add "Choosing a Time Series Database for High Cardinality Aggregations" article (#1617 )	2021-09-15 09:35:38 +03:00
Aliaksandr Valialkin	63fb5a5ca5	docs/vmalert.md: follow-up after `68721f6e7d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1608	2021-09-14 14:48:45 +03:00
Roman Khavronenko	eea6317d40	vmalert: support bearer token for datasource, remotewrite and remoteread (#1614 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1608	2021-09-14 14:48:43 +03:00
Dima Lazerka	5a256970d6	Remove port 3000 and add https to play-grafana link (#1616 ) * Remove port 3000 and add https to play-grafana link * Fix typo Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>	2021-09-14 14:27:16 +03:00
Aliaksandr Valialkin	1493461244	lib/storage: follow up after `00cbb099b6`	2021-09-14 14:23:02 +03:00
faceair	61a51f7c15	lib/storage: optimize convert multiple values regexp filter to composite tag filter (#1610 ) * lib/storage: optimize convert multiple values regexp filter to composite tag filter * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-09-14 14:23:01 +03:00
Aliaksandr Valialkin	5e5ce27df7	app/vmauth: do not log invalid auth tokens by default for security reasons The logging can be enabled by passing `-logInvalidAuthTokens` command-line flag to vmauth	2021-09-14 12:20:17 +03:00
Aliaksandr Valialkin	d3289bf276	docs/Single-server-VictoriaMetrics.md: link to cardinality limiter docs in vmagent	2021-09-13 21:27:25 +03:00
Aliaksandr Valialkin	a8f1b614e5	docs/vmagent.md: mention `out of order sample` errors, which are typically emitted by Thanos, Cortex or Prometheus	2021-09-13 19:36:09 +03:00
Aliaksandr Valialkin	a1b298a842	app/vminsert/netstorage: disable rerouting by default Production clusters work more stable with the disabled rerouting during rolling restarts and/or during spikes in time series churn rate. So it would be better disabling the rerouting by default. The re-routing can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert` nodes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165	2021-09-13 18:50:47 +03:00
Aliaksandr Valialkin	17f8ea6e15	README.md: follow-up after `7f0a8d4bdb`	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	2932f495f3	docs/CHANGELOG.md: document `5494bc02a6`	2021-09-13 17:14:45 +03:00
Roman Khavronenko	7df731ea91	vmalert: add flag to limit the max value for auto-resovle duration for alerts (#1609 ) * vmalert: add flag to limit the max value for auto-resovle duration for alerts The new flag `rule.maxResolveDuration` suppose to limit max value for alert.End param, which is used by notifiers like Alertmanager for alerts auto resolve. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1586	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	184e145570	docs: consistency renaming: Influx -> InfluxDB	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	101cdef0a7	app/vmui/Dockerfile-web: update Go builder from 1.16.7 to 1.17.1 and Alpine base image from 3.14.1 to 3.14.2	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	b721110902	docs/vmbackup.md: update the outdated link to vmbackupmanager	2021-09-13 14:17:53 +03:00
Roman Khavronenko	9dd121db36	vmalert: display extra filter labels in UI (#1613 )	2021-09-13 14:17:51 +03:00
Aliaksandr Valialkin	19516aedb5	app/vmselect/promql: use Prometheus-compatible label value formatting for `count_values` function	2021-09-13 13:48:18 +03:00
Aliaksandr Valialkin	b684624f67	lib/promscrape/discovery/docker: support host networking mode See https://github.com/prometheus/prometheus/issues/9116	2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin	6ed9f10da5	lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery See https://github.com/prometheus/prometheus/issues/8902	2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin	1f6082be3a	vendor: `make vendor-update`	2021-09-12 15:28:11 +03:00
Aliaksandr Valialkin	da85ec16c3	deployment/docker: update Go builder from v1.17.0 to v1.17.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.17.1+label%3ACherryPickApproved	2021-09-12 15:23:47 +03:00
Aliaksandr Valialkin	d90834da70	lib/promscrape: generate `scrape_timeout_seconds` metric per each scrape target in the same way as Prometheus 2.30 does See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin	279f37c9e7	lib/promscrape: `make fmt`	2021-09-12 13:35:21 +03:00
Aliaksandr Valialkin	6c97388dde	lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling See https://github.com/prometheus/prometheus/pull/8911	2021-09-12 13:35:20 +03:00
Aliaksandr Valialkin	09670479cd	lib/promscrape: reduce CPU usage for common case when calculating `scrape_series_added` metric Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. The main idea is to perform the calculations on scrape_series_added and series_limit only if the set of metrics exposed by the target has been changed. Scrape targets rarely change the set of exposed metrics, so this optimization should reduce CPU usage in general case.	2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin	2b64c1c95e	docs/Single-server-VictoriaMetrics.md: refer to relabeling section for vmagent This removes duplicate docs about additional relabeling actions supported by VictoriaMetrics components	2021-09-12 11:40:23 +03:00
Aliaksandr Valialkin	def0430318	docs/vmagent.md: sync with app/vmagent/README.md by running `make docs-sync`	2021-09-11 11:05:12 +03:00
Aliaksandr Valialkin	c339642858	lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric	2021-09-11 11:03:38 +03:00
Aliaksandr Valialkin	6d6cf1b6e0	lib/storage: verify that the tsidsFound contain the needed tsids in tests added at `f4dead529f`	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	5aaaa686a4	lib/promscrape: send stale markers for disappeared metrics like Prometheus does	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	e1632dc0df	docs/vmalert.md: typo fix in `Multitenancy` chapter	2021-09-10 17:57:43 +03:00
Aliaksandr Valialkin	c2f37f049b	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz\|",x=~"y\|"} Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:12:53 +03:00
Aliaksandr Valialkin	9b07cb2988	app/vmselect/promql: keep metric name in `rollup_candlestick` results, since they don't change the original series meaning Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600	2021-09-09 19:24:10 +03:00
Aliaksandr Valialkin	dca6f0f7de	app/vminsert/netstorage: remove the limit on the number of `-storageNode` addresses There is no any reasons to limit the number of `-storageNode` addresses to 255. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1599	2021-09-09 19:11:36 +03:00
Aliaksandr Valialkin	09edcfd21c	docs/CHANGELOG.md: typo fix	2021-09-09 18:58:03 +03:00
Aliaksandr Valialkin	d613a018c8	lib/promscrape: add ability to set `series_limit` and `stream_parse` options via relabeling This allows managing these options on a per-target basis. Typical use case: to manage these options for pods via Kubernetes annotations.	2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin	b64866e64c	lib/promscrape: add `scrape_` prefix to `job` and `target` labels exported by `promscrape_series_limit_rows_dropped_total` metric This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.	2021-09-09 17:31:04 +03:00
Aliaksandr Valialkin	986807ebd9	docs/vmagent.md: typo fix in `Relabeling` chapter	2021-09-09 16:39:34 +03:00
Denys Holius	9aac6acb6b	Bump alpine linux to latest (#1607 )	2021-09-09 16:30:17 +03:00
Aliaksandr Valialkin	75c3514c5c	lib/promrelabel: add `keep_metrics` and `drop_metrics` actions to relabeling rules These actions simlify metrics filtering. For example, - action: keep_metrics regex: 'foo\|bar\|baz' would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted. The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above: - action: keep_metrics regex: - foo - bar - baz	2021-09-09 16:25:09 +03:00
Aliaksandr Valialkin	708ebd59ad	docs/vmalert.md: `make docs-sync` after `21f022e5f0`	2021-09-09 16:25:08 +03:00
Aliaksandr Valialkin	2b19dd8601	docs/MetricsQL.md: add a link to VictoriaMetrics github	2021-09-08 00:15:34 +03:00
Aliaksandr Valialkin	d72593cca2	app/vmalert: document GroupAlerts This makes golint happy	2021-09-07 22:50:48 +03:00
Aliaksandr Valialkin	608ed19655	app/vmalert: follow-up after `21f022e5f0`	2021-09-07 22:45:47 +03:00
Roman Khavronenko	bb26aa9b1f	vmalert: add initial UI implementation (#1602 ) New UI pages: / - welcome page with API handlers list; /groups - list of all rules per group; /alerts - list of all active alerts; /groupID/alertID/status - status of the active alert;	2021-09-07 22:45:45 +03:00
dependabot[bot]	83356af4c7	build(deps): bump github.com/aws/aws-sdk-go from 1.40.34 to 1.40.37 (#1598 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.34 to 1.40.37. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.34...v1.40.37) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-09-07 22:32:16 +03:00
Aliaksandr Valialkin	9be7294331	docs/CHANGELOG.md: document `42e07cfaea`	2021-09-07 22:32:01 +03:00
mxlxm	42e07cfaea	reset deadline, fix #1562 . (#1597 ) * reset deadline, fix #1562. reset deadline before we put it back to pool. * make errcheck happy	2021-09-07 20:54:17 +03:00
Aliaksandr Valialkin	304661c547	docs/vmagent.md: add API path for Prometheus text exposition format	2021-09-07 16:15:28 +03:00
Aliaksandr Valialkin	c5e030dfc4	docs/Articles.md: add a link to case studies	2021-09-06 10:58:48 +03:00
Aliaksandr Valialkin	52b44945a0	docs/Articles.md: add an url to https://www.vultr.com/docs/install-and-configure-victoriametrics-on-debian	2021-09-03 11:50:23 +03:00
Aliaksandr Valialkin	9f816dfee8	docs/Single-server-VictoriaMetrics.md: fix a link multitenancy docs for cluster version in VictoriaMetrics	2021-09-02 17:44:22 +03:00
Aliaksandr Valialkin	a88b02287d	docs/CHANGELOG.md: cut v1.65.0	2021-09-01 17:12:44 +03:00
Roman Khavronenko	18313f3f8e	Cluster dashboard update (#1594 ) * dashboards: sync `vmagent` updates from master branch * dashboards: add new `Storage connection saturation` panel for cluster dashboard * dashboards: add new cluster alert for corresponding `Storage connection saturation` panel	2021-09-01 17:05:17 +03:00
Aliaksandr Valialkin	41af6a73fe	docs/FAQ.md: add questions on how to migrate data from various systems (Prometheus, InfluxDB, OpenTSDB, Graphite) to VictoriaMetrics	2021-09-01 16:48:05 +03:00
Aliaksandr Valialkin	5c58c8d55e	docs: update `-help` output for victoria-metrics and vmagent after `f77dde837a`	2021-09-01 16:35:18 +03:00
Aliaksandr Valialkin	30d2876c6e	.github/dependabot.yml: increase check intervals for gomod and docker ecosystems from daily to weekly Daily checks are too verbose and result into too many automatic pull requests and commits	2021-09-01 16:08:31 +03:00
Roman Khavronenko	be3e31a574	dashboards: bump vmagent version requirement	2021-09-01 16:08:17 +03:00
Aliaksandr Valialkin	c4df601f43	lib/promscrape: add the ability to limit the number of unique series per each scrape target The number of series per target can be limited with the following options: * Global limit with `-promscrape.maxSeriesPerTarget` command-line option. * Per-target limit with `max_series: N` option in `scrape_config` section. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561	2021-09-01 16:08:12 +03:00
dependabot[bot]	715977fc58	build(deps): bump github.com/aws/aws-sdk-go from 1.40.33 to 1.40.34 (#1591 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.33 to 1.40.34. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.33...v1.40.34) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-09-01 12:54:54 +03:00
dependabot[bot]	cea8e0c20a	build(deps): bump google.golang.org/api from 0.55.0 to 0.56.0 (#1590 ) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.55.0 to 0.56.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.55.0...v0.56.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-09-01 12:54:54 +03:00
dependabot[bot]	aac67a9492	build(deps): bump cloud.google.com/go/storage from 1.16.0 to 1.16.1 (#1589 ) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.16.0 to 1.16.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/master/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.16.0...storage/v1.16.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-09-01 12:54:54 +03:00
Aliaksandr Valialkin	835efb111e	docs: `make docs-sync`	2021-09-01 12:25:17 +03:00
Aliaksandr Valialkin	816d4b6095	docs/CHANGELOG.md: document `eff940aa76`	2021-09-01 12:25:11 +03:00
Aliaksandr Valialkin	22bb18a449	docs/CHANGELOG.md: document `7c70dcbe3b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471	2021-09-01 12:25:04 +03:00
Roman Khavronenko	af8c1feddb	Single dashboards upd (#1593 ) * dasbhoard: replace `null` datasources null datasource value may confuse Grafana and make it drop panel query in some versions. * docker: bump grafana image version * dashboards: add URL variable selector to vmagent dashboard * dashboards: add new panel `Remote write connection saturation` to vmagent dashboard * alerts: add new alert for `Remote write connection saturation` panel of vmagent dashboard * dashboards: add "Logging rate" panel to vmagent dashboard	2021-09-01 12:24:55 +03:00
Roman Khavronenko	8962a50478	vmctl: update README and flags description (#1588 ) The purpose of update is to make README and flags description more clear to the reader. Especially, show that vm-account-id flag is required for clustered version of VM.	2021-09-01 12:22:58 +03:00
Roman Khavronenko	1cf4f5a715	Vmalert extra params (#1587 ) * vmalert: allow extra GET params in datasource package ExtraParams will be added as GET params to every HTTP request made by datasource. The `roundDigits` param, for example, was substituted by corresponding extra param. * vmalert: add nocache=1 param for replay process The `nocache=1` param is VictoriaMetrics specific parameter which prevents it from caching and boundaries aligning for queries. We set it to avoid cache pollution in `replay` mode and also to avoid unnecessary time range boundaries alignment. * vmalert: mention nocache=1 in replay description * vmalert: fix bug with unused param	2021-09-01 12:20:01 +03:00
Nikolay	210c76b51e	adds external_labels per group for vmalert (#1485 ) * adds external_label per group for vmalert https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471	2021-09-01 12:19:49 +03:00
Roman Khavronenko	1cb7037fc8	Vmalert metrics update (#1580 ) * vmalert: remove `vmalert_execution_duration_seconds` metric The summary for `vmalert_execution_duration_seconds` metric gives no additional value comparing to `vmalert_iteration_duration_seconds` metric. * vmalert: update config reload success metric properly Previously, if there was unsuccessfull attempt to reload config and then rollback to previous version - the metric remained set to 0. * vmalert: add Grafana dashboard to overview application metrics * docker: include vmalert target into list for scraping * vmalert: extend notifier metrics with addr label The change adds an `addr` label to metrics for alerts_sent and alerts_send_errors to identify which exact address is having issues. The according change was made to vmalert dashboard. * vmalert: update documentation and docker environment for vmalert's dashboard Mention Grafana's dashboard in vmalert's README in a new section #Monitoring. Update docker-compose env to automatically add vmalert's dashboard. Update docker-compose README with additional info about services.	2021-09-01 12:19:34 +03:00
Roman Khavronenko	434f33d04d	Cluster sync master changes (#1592 ) * docker: add README for docker compose env * docker: add vmalert Grafana dashboard	2021-09-01 10:25:07 +03:00
Aliaksandr Valialkin	0b5f30b4f9	vendor: `make vendor-update`	2021-08-31 12:00:06 +03:00
Aliaksandr Valialkin	9723ac30dd	docs/Single-server-VictoriaMetrics.md: remove outdated link to VictoriaMetrics wiki VictoriaMetrics wiki became outdated after publishing all the docs at https://docs.victoriametrics.com	2021-08-31 11:53:38 +03:00
Aliaksandr Valialkin	b3d240f7b4	docs/CHANGELOG.md: add a link to Prometheus staleness tracking	2021-08-31 11:53:38 +03:00
dependabot[bot]	9a1e079f92	build(deps): bump github.com/aws/aws-sdk-go from 1.40.30 to 1.40.33 (#1582 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.30 to 1.40.33. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.30...v1.40.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-31 11:53:38 +03:00
dependabot[bot]	7a53e2d093	build(deps): bump google.golang.org/api from 0.54.0 to 0.55.0 (#1583 ) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.54.0 to 0.55.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.54.0...v0.55.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-31 11:53:38 +03:00
dependabot[bot]	75596ad423	build(deps): bump github.com/klauspost/compress from 1.13.4 to 1.13.5 (#1584 ) Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.13.4 to 1.13.5. - [Release notes](https://github.com/klauspost/compress/releases) - [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml) - [Commits](https://github.com/klauspost/compress/compare/v1.13.4...v1.13.5) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-31 11:53:38 +03:00
Aliaksandr Valialkin	146c14d879	lib/promscrape/discovery/kubernetes: return back support `role: endpointslices`, since it is used by VictoriaMetrics operator This is a follow up commit after `31b42b30b6`	2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin	3788d4f4eb	app/vmselect: show useful endpoints when requested `/select/<accountID>/` page	2021-08-29 12:05:14 +03:00
Aliaksandr Valialkin	18d7adf731	lib/protoparser/opentsdb: follow-up after `8ee75ca45a`	2021-08-29 11:50:01 +03:00
envzhu	00dddfe02f	lib/protoparser/opentsdb: accept multiple spaces between fields in a row as a deliminator. (#1575 )	2021-08-29 11:50:00 +03:00
Aliaksandr Valialkin	ca61d7c82b	lib/promscrape/discovery/kubernetes: rename `role: endpointslices` to `role: endpointslice` to be consistent with Prometheus See `2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)`	2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin	327034b54f	lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for `role: ingress` and `role: endpointslices` This should fix service discovery for these roles in Kubernetes v1.22 and newer versions. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122 The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205	2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin	0e809bd4b6	docs/Single-server-VictoriaMetrics.md: mention that downsampling doesnt improve query performance on high churn rate	2021-08-27 18:51:28 +03:00
Aliaksandr Valialkin	39bb6bdd79	app/vmselect/promql: add `quantile("phiLabel", phi1, ..., phiN, q)` aggregate function to MetricsQL See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573	2021-08-27 18:40:25 +03:00
Aliaksandr Valialkin	14d4b5aa83	app/vmselect: add `-search.disableAutoCacheReset` command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570	2021-08-27 17:17:43 +03:00
Aliaksandr Valialkin	6e5864b014	docs/{vmgateway,vmbackupmanager}: mention that enterprise binaries are free for download and evaluation	2021-08-27 14:53:48 +03:00
Aliaksandr Valialkin	adf71a415e	docs: link to `active time series`, `churn rate` and `high cardinality` questions	2021-08-27 14:45:36 +03:00
Aliaksandr Valialkin	1a7646c142	docs/CHANGELOG.md: document the bugfix for possible timeout error in vmbackupmanager when making snapshots See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571	2021-08-27 13:04:33 +03:00
Aliaksandr Valialkin	125a94e6ca	docs: mention that enterprise binaries can be downloaded and evaluated for free	2021-08-27 12:48:51 +03:00
Aliaksandr Valialkin	8845dc63db	vendor: `make vendor-update`	2021-08-26 09:43:01 +03:00
Aliaksandr Valialkin	102ab795f8	docs/vmagent.md: document the ability to load scrape configs from multiple files See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559	2021-08-26 09:13:54 +03:00
Aliaksandr Valialkin	7fdb4db73d	lib/promscrape: add ability to load scrape configs from multiple files See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559	2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin	dd96792a43	vendor: `make vendor-update`	2021-08-25 13:42:34 +03:00
Aliaksandr Valialkin	8d843d0754	docs/CHANGELOG.md: document `48f33d098b`	2021-08-25 13:32:05 +03:00
benclive	76816a0193	Remove trailing slash for URLPrefixes with specific path (#1554 )	2021-08-25 13:32:04 +03:00
Aliaksandr Valialkin	874660a4ae	docs/Cluster-VictoriaMetrics.md: mention that the `-replicationFactor` at `vmselect` is an optional parameter	2021-08-25 13:10:31 +03:00
Aliaksandr Valialkin	4a2d7aec7f	lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config	2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin	b885bd9b7d	lib/{mergeset,storage}: improve the detection of the needed free space for background merge This should prevent from possible out of disk space crashes during big merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560	2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin	ae8ec78c63	docs/FAQ.md: add more entries for frequently asked questions The following topics are covered: * Active time series * High cardinality * High churn rate * Slow inserts	2021-08-24 11:34:31 +03:00
Aliaksandr Valialkin	9b39e078c0	docs/MetricsQL.md: typo fix: histogram_qunatile -> histogram_quantile	2021-08-23 23:08:28 +03:00
Aliaksandr Valialkin	335de30083	app/vmselect/promql: `make fmt` after `0078486ea7`	2021-08-23 23:05:34 +03:00
Aliaksandr Valialkin	3eca49c4a6	docs/MetricsQL.md: fix the indentation for `median` function	2021-08-23 12:04:43 +03:00
Aliaksandr Valialkin	a4948d92b5	docs/MetricsQL.md: typo fix: convesions->conversions	2021-08-23 12:01:34 +03:00
Aliaksandr Valialkin	8b9dc45c3c	docs/MetricsQL.md: typo fixes	2021-08-23 12:00:17 +03:00
Aliaksandr Valialkin	5917c72ddd	docs/MetricsQL.md: rehaul the documentation on MetricsQL * Document all the functions supported by MetricsQL, including PromQL functions * Group functions by their type: rollup functions, transform functions, label manipulation functions and aggregate functions. * Document implicit query transformations.	2021-08-23 11:46:30 +03:00
Aliaksandr Valialkin	40b06e84f8	app/vmselect/promql: rename `sign()` function to `sgn()` in order to be consistent with Prometheus See https://github.com/prometheus/prometheus/pull/8457 for details.	2021-08-23 11:46:29 +03:00
Aliaksandr Valialkin	8493159eed	deployment/docker: update Go builder from Go1.16.7 to Go1.17.0 This improves data ingestion and query performance by up to 5% according to benchmarks. See https://go.dev/blog/go1.17	2021-08-21 22:22:31 +03:00
Aliaksandr Valialkin	67bc407747	lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets Store the scraped response body instead of storing the parsed and relabeld metrics. This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics. This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics. This should also reduce CPU usage, since the marshaling of the parsed and relabeld metrics has been substituted by response body copying. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin	ff4c7c1a3d	docs/vmalert.md: run `make docs-sync` after `9ee3d0378f`	2021-08-21 20:25:26 +03:00
Aliaksandr Valialkin	388e07b37f	docs/CHANGELOG.md: document `9ee3d0378f`	2021-08-21 20:23:22 +03:00
Roman Khavronenko	0c2284b95f	vmalert: add flag `disableAlertgroupLabel` for disabling extra label added to series (#1534 ) The new label added in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/611 may negatively impact deduplication in Alertmanager. The new flag supposed to give an option to disable adding this label. To enable flag just add `-disableAlertgroupLabel` to binary execution command. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1532	2021-08-21 20:23:22 +03:00
Alexander Rickardsson	9e2e9d83a5	vmalert: accept http.StatusOK for remotewrite (#1550 )	2021-08-21 20:23:22 +03:00
Aliaksandr Valialkin	0ff3e04ed4	vendor: `make vendor-update`	2021-08-21 20:13:44 +03:00
Aliaksandr Valialkin	91534057a3	app/vmselect/prometheus: do not extend `[d]` to the detected interval between samples for `first_over_time(m[d])` This is for the sake of consistency with similar change for the last_over_time(m[d]) at `a724229b5d`	2021-08-21 19:56:56 +03:00
dependabot[bot]	959a667aa5	build(deps): bump github.com/aws/aws-sdk-go from 1.40.25 to 1.40.26 (#1551 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.25 to 1.40.26. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.25...v1.40.26) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-21 19:56:55 +03:00
Aliaksandr Valialkin	c3b24882a7	lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape This will make timestamps for stale markers more consistent for timestamps for other samples	2021-08-19 14:19:54 +03:00
Aliaksandr Valialkin	3454f25e0f	docs/CHANGELOG.md: document `b5d6a0e499`	2021-08-19 14:07:00 +03:00
Roman Khavronenko	1ccb77904b	vmselect: update `vm_request_duration_seconds` value when request fails (#1537 ) Before, metric `vm_request_duration_seconds` was update only on successful attempts which could be misleading. For example, timeout errors on netstorage request may be not accounted in the metric and won't be visible on dashboards. Using `defer` statement to update the metric after query arguments validation may improve the situation.	2021-08-19 14:07:00 +03:00
dependabot[bot]	5a8ee065b1	build(deps): bump github.com/aws/aws-sdk-go from 1.40.22 to 1.40.25 (#1548 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.22 to 1.40.25. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.22...v1.40.25) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-19 14:07:00 +03:00
Aliaksandr Valialkin	2e1eb33bfd	docs/CHANGELOG.md: clarify the change, which adds `-search.noStaleMarkers` command-line flag	2021-08-19 13:55:43 +03:00
Aliaksandr Valialkin	ee1f3414d1	app/vmselect/promql: do not override `[d]` at `last_over_time(m[d])` if `[d]` is smaller than `scrape_interval` Since most users do not expect the overriding of explicitly set `[d]`.	2021-08-19 10:33:10 +03:00
Aliaksandr Valialkin	47a37b8cfc	docs/CHANGELOG.md: cut v1.64.1 This is mostly bugfix release, which includes fixes for staleness handling and a security update for Alpine base image	2021-08-18 22:07:05 +03:00
Aliaksandr Valialkin	8ee575dee9	lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does	2021-08-18 22:00:46 +03:00
Max Golionko	738741ab0d	rename group for cluster (#1546 ) rename group for cluster, so that they not overlap when you have vmsingle and vmcluster deployed alongside	2021-08-18 16:03:04 +03:00
Aliaksandr Valialkin	c9c583ff5b	vendor: update github.com/valyala/gozstd from v1.11.0 to v1.12.0 This should improve query scalability on systems with big number of CPU cores	2021-08-18 14:58:08 +03:00
Aliaksandr Valialkin	355690a719	docs/CHANGELOG.md: document `06bf21c21b`	2021-08-18 14:01:42 +03:00
Aliaksandr Valialkin	5d92fafc40	app/vmselect: add `-search.noStaleMarkers` command-line flag for disabling stale markers handling in queries This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin	f21fad53b4	lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag This option can be useful when vmagent consumes too much additional memory for staleness markers functionality and when staleness markers aren't needed.	2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin	3ba7a875f3	deployment/docker: upgrade Alpine base docker image from v3.14.0 to v3.14.1 See https://www.alpinelinux.org/posts/Alpine-3.14.1-released.html This fixes https://vuldb.com/?source_cve.180051 See also https://vuldb.com/?id.180051 and https://snyk.io/vuln/SNYK-ALPINE314-APKTOOLS-1533752	2021-08-18 11:04:03 +03:00
Aliaksandr Valialkin	49886ecbc8	app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541	2021-08-17 13:22:15 +03:00
Aliaksandr Valialkin	130a5cab7e	docs/Single-server-VictoriaMetrics.md: mention that vmctl can migrate data from OpenTSDB to VictoriaMetrics	2021-08-17 11:17:38 +03:00
Aliaksandr Valialkin	38065bec7b	app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability	2021-08-17 11:00:58 +03:00
Aliaksandr Valialkin	db34c40aec	lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509	2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin	fe8c462044	app/vmalert: mention -remoteWrite.disablePathAppend in the description for -remoteWrite.url	2021-08-16 15:23:39 +03:00
Aliaksandr Valialkin	21974cb571	app/vmalert: follow-up for `2400f85761`	2021-08-16 15:20:35 +03:00
Aliaksandr Valialkin	a47c1a734a	docs/CHANGELOG.md: update urls to Prometheus 2.29 release Previously these urls were pointing to rc0 release	2021-08-16 14:58:05 +03:00
Aliaksandr Valialkin	f05e827757	docs/CHANGELOG.md: typo fix: satureated -> saturated	2021-08-16 14:58:05 +03:00
Alexander Rickardsson	d27dc3721b	vmalert: enable configuring explicit path (#1536 ) * vmalert: allow to disable automatically added path to remote write address via disablePathAppend flag * docs: update docs to include remoteWrite.disablePathAppend	2021-08-16 14:58:05 +03:00
dependabot[bot]	a9813fa032	build(deps): bump github.com/aws/aws-sdk-go from 1.40.21 to 1.40.22 (#1539 ) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.21 to 1.40.22. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.21...v1.40.22) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-16 14:58:05 +03:00
dependabot[bot]	2a24e0740f	build(deps): bump google.golang.org/api from 0.53.0 to 0.54.0 (#1538 ) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.53.0 to 0.54.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.53.0...v0.54.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-08-16 14:58:05 +03:00
Aliaksandr Valialkin	75a0345215	docs/CHANGELOG.md: cut v1.64.0	2021-08-15 23:52:24 +03:00
Aliaksandr Valialkin	5f13c519ee	lib/promscrape: restore red highlighting for DOWN targets at /targets page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461	2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin	175727dace	docs/CHANGELOG.md: mention the bugfix when more than 27 time series are selected at /vmui	2021-08-15 15:11:13 +03:00
Aliaksandr Valialkin	b09b035d3e	docs/CHANGELOG.md: mention that VMUI automatically fills Server URL field Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1506	2021-08-15 14:45:59 +03:00
Artem Navoiev	c0420634ff	add dependency chekcs for (#1535 ) - ruby (for docs) - gomod for monorepo - npm for vmui - gomod go small webserver in vmui	2021-08-15 14:45:53 +03:00
Aliaksandr Valialkin	48920bdef8	app/vmagent/remotewrite: expose vmagent_remotewrite_send_duration_seconds_total metric This metric can be used for determining high saturation of every connection to remote storage with an alerting query `rate(vmagent_remotewrite_send_duration_seconds_total) > 0.9s`. This query triggers when a connection is satureated by more than 90%	2021-08-15 13:34:07 +03:00
Aliaksandr Valialkin	5420c3d967	app/vmselect/promql: drop staleness marks before calling rollupConfig.Do This allows dropping staleness marks only once and then calculate multiple rollup functions on the result. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-15 13:22:26 +03:00
Aliaksandr Valialkin	6c4c54eaad	Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions" This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346. It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc	2021-08-15 13:22:24 +03:00
Aliaksandr Valialkin	af4a306d7b	app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values, which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-14 12:45:31 +03:00
Aliaksandr Valialkin	7d0e64dcc0	vendor: `make vendor-update`	2021-08-13 13:03:03 +03:00
Aliaksandr Valialkin	c1f81f08d4	all: add support for Prometheus staleness markers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin	7feb62eea9	docs/Cluster-VictoriaMetrics.md: meniton that vmagent can be used for replicating the data among multiple clusters Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491	2021-08-12 12:47:08 +03:00
Aliaksandr Valialkin	60aa9cc1e6	vendor: update github.com/klauspost/compress from v1.13.1 to v1.13.4	2021-08-12 12:40:49 +03:00
Denys Holius	1491767a9e	added guide for HA monitoring setup in K8s via VM Cluster (#1523 ) * added guide for HA monitoring setup in K8s via VM Cluster * fixed missed divs * fixed different typos * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin	b35ae791f1	app/vmselect: `make vmui-update` after the commit 4ae14df864a7e327955f44941295a286175423b3	2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin	f60ff85dbe	app/vmui: actualize Dockerfiles	2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin	9eb828b2c2	app/vminsert: add vm_rpc_send_duration_seconds_total metric per each `vminsert->vmstorage` link This metric is useful for determining high link saturation with the following alerting rule: rate(vm_rpc_send_duration_seconds_total) > 0.9s	2021-08-11 11:42:33 +03:00
Aliaksandr Valialkin	3df6550153	docs/Cluster-VictoriaMetrics.md: update `-help` output for cluster components after the `d375d9b878`	2021-08-11 11:39:09 +03:00
Aliaksandr Valialkin	518d0aba18	docs: `make docs-sync` after `e0ee69797d`	2021-08-11 10:54:28 +03:00
Aliaksandr Valialkin	90efb5831b	lib/envflag: add a link to docs for -envflag.enable	2021-08-11 10:32:40 +03:00
Aliaksandr Valialkin	3823dab820	deployment/dm: update Go builder from Go1.16.6 to Go1.16.7 See https://github.com/golang/go/issues?q=milestone%3AGo1.16.7+label%3ACherryPickApproved	2021-08-06 12:16:54 +03:00
Yury Molodov	aca2cb245e	vmui: fix layout and add server url by default (#1519 ) * fix: change layout for correctly display big query * fix: set default server from url * fix: change get default server url	2021-08-06 12:16:53 +03:00
Aliaksandr Valialkin	d5ca07bd71	docs/CHANGELOG.md: document new metrics added to vmalert at `7416fdaa8b`	2021-08-05 10:12:53 +03:00
Roman Khavronenko	d5ba8248cc	vmalert: expose new metrics for tracking number of produced samples during last evaluation (#1518 ) * vmalert: expose new metrics for tracking number of produced samples during last evaluation Two new metrics were added to track the number of samples produced during the last evaluation: * vmalert_recording_rules_last_evaluation_samples * vmalert_alerting_rules_last_evaluation_samples The gauge type is used to remain consistent with Prometheus metric `prometheus_rule_group_last_evaluation_samples` which is on the group level. However, the counter type was considered as well. Two metrics instead of one are used to make it easier to separate recording and alerting rules. It is likely, number of samples produced by recording rules is more important so people will refer to it more frequently. The expected usage of the new metric is the following: ``` - alert: RecordingRuleReturnsEmptyResults expr: sum(vmalert_recording_rules_last_evaluation_samples) by(recording) < 1 annotations: summary: Recording rule {{$labels.recording}} returns empty results. Please verify expression correctness. ``` Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1494 * vmalert: rename `vmalert_alerts_error` to `vmalert_alerting_rules_error` to remain consistent with recording rules metrics	2021-08-05 10:02:35 +03:00
Aliaksandr Valialkin	13d438d808	app/vmagent: typo fix in the description for `-remoteWrite.queues`	2021-08-05 10:00:58 +03:00
Aliaksandr Valialkin	b877538622	app/vmagent: follow-up after `fe445f753b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491	2021-08-05 09:51:00 +03:00
Omar Ghader	fe445f753b	feature: Add multitenant for vmagent (#1505 ) * feature: Add multitenant for vmagent * Minor fix * Fix rcs index out of range * Minor fix * Fix multi Init * Fix multi Init * Fix multi Init * Add default multi * Adjust naming * Add TenantInserted metrics * Add TenantInserted metrics * fix: remove unused metrics for vmagent * fix: remove unused metrics for vmagent Co-authored-by: mghader <marc.ghader@ubisoft.com> Co-authored-by: Sebastian YEPES <syepes@gmail.com>	2021-08-05 09:44:29 +03:00
Qifei Wan	095bb90879	app/vmalert: update config state metrics if config parsed failed (#1507 )	2021-08-03 16:12:48 +03:00
Aliaksandr Valialkin	44bf4cabea	docs/Articles.md: actualize links and re-order some links	2021-08-03 16:11:30 +03:00
Aliaksandr Valialkin	77bb9e1656	lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 See https://github.com/prometheus/prometheus/pull/8978	2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin	336a2aa2e0	lib/promscrape/discovery/ec2: add `__meta_ec2_availability_zone_id` label as Prometheus 2.29 does	2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin	60cfa5f100	app/vmselect/promql: add `present_over_time(m[d])` function, which will be available starting from Prometheus 2.29.0 See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097	2021-08-03 12:21:53 +03:00
Roman Khavronenko	51329d9e1e	docs: add "Scaling to trillions of metric data points" to articles (#1517 )	2021-08-03 11:14:57 +03:00
wusphinx	511e5c2e68	Update TimeSelector.tsx (#1515 ) delete garbled code	2021-08-03 11:14:56 +03:00
Roman Khavronenko	d63842cdbe	Cluster alerts (#1513 ) * alerts: move `ProcessNearFDLimits` to `vm-health` group since it is relevant for all services * alerts: add new `TooHighMemoryUsage` alerting rule	2021-08-02 17:54:24 +03:00
Nikolay	3f3ad13753	adds /rules and /alerts api for grafana (#1504 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-08-02 17:29:49 +03:00
Roman Khavronenko	e127dcf013	docs: mention "Push Prometheus metrics to VictoriaMetrics or other exporters" article (#1511 )	2021-08-02 17:23:45 +03:00
Roman Khavronenko	8e15c96004	docs: fix indentation for guide articles (#1512 )	2021-08-02 17:17:24 +03:00
Aliaksandr Valialkin	d9810a7403	docs/CaseStudies.md: typo fix: hed->had	2021-07-30 18:49:17 +03:00
Aliaksandr Valialkin	c2df339eb2	docs/CHANGELOG.md: typo fix	2021-07-30 12:35:28 +03:00
Aliaksandr Valialkin	3389f1e474	docs/CHANGELOG.md: document `d05cac6c98` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486	2021-07-30 12:26:34 +03:00
Aliaksandr Valialkin	99004a6a40	app/vmselect/netstorage: unpack time series data in mostly local big chunks This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples	2021-07-30 12:26:33 +03:00
Aliaksandr Valialkin	c473d8ffe1	li/storage: re-use the per-day inverted index search code for searching in global index This allows removing a big pile of outdated code for global index search. This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486	2021-07-30 10:28:20 +03:00
Aliaksandr Valialkin	cbb81c2ce9	app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17	2021-07-30 10:28:19 +03:00
Aliaksandr Valialkin	b709fa387a	app/vmselect: follow-up for `ed95bc9531` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493	2021-07-29 09:48:47 +03:00
arnoldyahad	ed95bc9531	Add case prometheus/rules for grafana 8 (#1502 )	2021-07-29 06:15:35 +03:00
assassins	6ab0001a1f	Performance optimization (#1481 ) There are redundant steps	2021-07-28 19:29:22 +03:00
Aliaksandr Valialkin	c582d95cf0	docs: remove SampleSizeCalculations.md, since it is outdated and no longer used There was a reference to this doc from the old victoriametrics.com site	2021-07-28 19:25:45 +03:00
Denys Holius	113df2f3b7	Added new guide for monitoring k8s via VictoriaMetrics cluster (#1476 ) * renamed and moved screenshots * fixed cluster guide, updated helm chart versions, added values.yaml for vm single * renamed guide files * fixed typo * add some fixes * fixed typos,added guide k8s-monitoring-via-vm-cluster * added fixes for yamls	2021-07-28 18:06:50 +03:00
Aliaksandr Valialkin	49bf3abf67	app/vmselect: follow-up for `626073bca8` * Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries * Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes * Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md	2021-07-28 18:00:04 +03:00
匠心零度	626073bca8	protection vmselect ,avoid metrics point too much let vmselect cup load very, very high (#1478 ) * protection vmselect…… * protection vmselect…… * protection vmselect…… * All checks have failed,fix Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-07-28 14:39:35 +03:00
Aliaksandr Valialkin	857fa4e28a	docs/CHANGELOG.md: document `9d45b46f4c`	2021-07-27 12:39:10 +03:00
Nikolay	6d47e750be	adds check for region with custom s3 endpoint (#1465 )	2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin	1950f57316	lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-27 12:03:31 +03:00
Aliaksandr Valialkin	92628f9f07	lib/promrelabel: add tests for verifying that regex works as expected in single quotes and double quotes	2021-07-27 10:53:03 +03:00
Aaron France	3c22b5c41e	fix: typo in metrics.md docs	2021-07-27 10:53:02 +03:00
Aliaksandr Valialkin	6f70a92452	vendor: `make vendor-update`	2021-07-26 16:01:51 +03:00
Aliaksandr Valialkin	5d255846ac	all: add `go:build` lines for Go1.17 See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin	3921d8afae	app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel	2021-07-26 15:50:45 +03:00
Aliaksandr Valialkin	c857e05604	lib/promscrape: add missing whitespace at /targets page before `up` word	2021-07-26 12:23:06 +03:00
Aliaksandr Valialkin	c3e6ce1db9	app/vmselect: `make vmui-update` after `a91d41f12a`	2021-07-26 10:32:01 +03:00
Aliaksandr Valialkin	349d45bbbe	docs/Articles.md: add links to new articles	2021-07-23 21:09:00 +03:00
Yury Molodov	401de2dca4	Vmui/query editor (#1472 ) * fix: move request button to server input * feat: add switch for query autocomplete * refactor: rename state for popover open * feat: add detect os by userAgent * fix: change hotkey to run query for mac * fix: change detect mac os * fix: change div to span inside Typography Co-authored-by: yury <yurymolodov@victoriametrics.com>	2021-07-23 21:08:58 +03:00
Aliaksandr Valialkin	b047feeb8b	app/vmselect/promql: properly handle `(a op b) default N` if `(a op b)` returns NaN series The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.	2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin	b92702f6d5	app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores	2021-07-16 00:35:36 +03:00
Aliaksandr Valialkin	df117f85bd	app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries or queries that take too much cpu time.	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	376af3c956	lib/workingsetcache: switch from split cache to full cache after the cache size exceeds 95% of split capacity Previously the switch occurred when the cache size becomes 100% of its capacity. The cache size could never reach 100% capacity. This could prevent from switching from the split cache to full cache, thus reducing the cache effectiveness.	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	5830ce2706	app/vmselect/netstorage: add `-search.maxSamplesPerSeries` command-line option for limiting the number of samples a query can process per each series This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	6c42db87a8	app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems	2021-07-15 15:40:53 +03:00
Aliaksandr Valialkin	3059e4feec	app/vmui/README.md: typo fix: naviate->navigate	2021-07-15 15:02:56 +03:00
Aliaksandr Valialkin	f847677513	docs/CHANGELOG.md: typo fix: suffxies->suffixes	2021-07-15 15:02:55 +03:00
Aliaksandr Valialkin	4ec57102e6	docs/CHANGELOG.md: cut v1.63.0	2021-07-15 14:02:45 +03:00
Aliaksandr Valialkin	c1fb991f56	vendor: `make vendor-update`	2021-07-15 12:57:10 +03:00
Aliaksandr Valialkin	9d3f9da5ad	lib/storage: make sure the second call to DeduplicateSamples and deduplicateSamplesDuringMerge doesnt change samples	2021-07-15 12:18:38 +03:00
Aliaksandr Valialkin	e992754e79	lib/storage: remove cache directory if it contains reset_cache_on_startup file See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447	2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin	ee7e9795ec	docs: clarify why spare CPU and RAM resources are needed in `capacity planning`	2021-07-13 15:49:38 +03:00
Aliaksandr Valialkin	03f1c0b58a	docs/Cluster-VictoriaMetrics.md: clarify the docs about the needed values for `-dedup.minScrapeInterval` at `vmselect` during replication when the data is pushed from HA pair	2021-07-13 15:29:43 +03:00
Aliaksandr Valialkin	e6edb85fa2	lib/httpserver: add `is_set` label to `flag` metrics This label allows determining the set flags with the query `flag{is_set="true"}`	2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin	23e6bc8a31	vendor: `make vendor-update`	2021-07-13 14:30:39 +03:00
Aliaksandr Valialkin	aecd744139	deployment/docker: update Go builder from v1.16.5 to v1.16.6 Ths Go release has the following bugfixes: https://github.com/golang/go/issues?q=milestone%3AGo1.16.6+label%3ACherryPickApproved	2021-07-13 14:25:35 +03:00
Aliaksandr Valialkin	9add9d86a6	app/vmselect/promql: duration handling improvements in MetricsQL queries - Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600 - Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])	2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin	51cd19d2e3	lib/storage: reset perKeyMisses stats less frequently This should reduce CPU usage for queries executed with intervals higher than 30 seconds	2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin	3f705fe8d7	lib/storage: properly limit the size of `storage/date_metricID` cache	2021-07-12 14:25:28 +03:00
Aliaksandr Valialkin	390a31ccfa	docs/CHANGELOG.md: document the change from `bfba4c28a4`	2021-07-12 12:42:42 +03:00
Aliaksandr Valialkin	d98e22fe50	app/vmalert: accept Prometheus-like durations in `interval` config option inside `group` section	2021-07-12 12:36:22 +03:00
Aliaksandr Valialkin	253fe7699c	docs: update http://slack.victoriametrics.com to https://slack.victoriametrics.com	2021-07-12 10:58:19 +03:00
Aliaksandr Valialkin	ef3c58d7a3	lib/storage: properly determine when the deduplication is needed in needsDedup Previously needsDedup() could return true if the de-duplication wasn't needed for the following case: d < interval / \ \| v \| v \| interval interval Now it properly returns false for this case	2021-07-12 10:54:51 +03:00
Aliaksandr Valialkin	f5fa177141	Revert "app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui" This reverts commit 885a79def6799f288e14df05b35a12569659ab85. Reason for revert: Grafana doesn't allows accessing /select/<accountID>/prometheus/vmui :(	2021-07-12 09:08:43 +03:00
Aliaksandr Valialkin	ddaa12050d	app/vmselect: expose vmui at /select/<accountID>/prometheus/vmui additionally to /select/<accountID>/vmui The /select/<accountID>/prometheus/vmui is needed for accessing via server-side Prometheus datasource for Grafana. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-10 12:52:25 +03:00
Aliaksandr Valialkin	0b98f6c7ff	app/vmselect: expose vmui at `/vmselect/<accountID>/vmui/` instead of `/vmselect/<accountID>/prometheus/vmui/` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-10 12:32:21 +03:00
Aliaksandr Valialkin	98e049ba6d	app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-09 17:13:51 +03:00
Aliaksandr Valialkin	9900a1f563	docs: clarify what does "workload" mean in capacity planning docs	2021-07-09 12:50:51 +03:00
Aliaksandr Valialkin	2c5e1cd893	app/vmselect: move web ui from /ui to /select/<accountID>/prometheus/ui This way the UI is available for every tenant (aka accountID) and the UI can automatically determine the needed per-tenant datasource path from page referer. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413	2021-07-08 13:14:50 +03:00
Aliaksandr Valialkin	acb7a95c64	app/vmselect: follow-up after `aa11ef6d3b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4	2021-07-07 17:45:09 +03:00
tony	aa11ef6d3b	add vmui for vmselect component (#1431 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-07-07 17:04:23 +03:00
Aliaksandr Valialkin	6f8dd85deb	docs/Cluster-VictoriaMetrics.md: improve capacity planning recommendations	2021-07-07 16:21:59 +03:00
Aliaksandr Valialkin	a374ede8e4	vendor: `make vendor-update`	2021-07-07 16:10:41 +03:00
Aliaksandr Valialkin	a2ad6644b9	vendor: update github.com/VictoriaMetrics/metrics from v1.17.2 to v1.17.3	2021-07-07 16:10:40 +03:00
Aliaksandr Valialkin	41754e12f8	lib/mergeset: cache indexBlock items only on the second request This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.	2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin	9c19719ad6	app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms	2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin	ceda2b1df4	lib/httpserver: print full requestURI in httpserver.Errorf This should simplify debugging.	2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin	4e87638877	docs: clarify `capacity planning` docs	2021-07-07 12:48:07 +03:00
Aliaksandr Valialkin	9826f7c1be	lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) This should reduce the cache size and memory usage for the indexdb/dataBlocks cache	2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin	74ace9340d	lib/storage: periodically reset prefetchedMetricIDs cache in order to limit its size under high churn rate	2021-07-07 10:59:39 +03:00
Roman Khavronenko	ce3f087d46	alerts: sync alert expression for `DiskRunsOutOfSpaceIn3Days` with dashboard (#1435 )	2021-07-07 00:47:08 +03:00
Aliaksandr Valialkin	a846febc89	Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" This reverts commit `7c6d3981bf`. Reason for revert: high contention at bucket16Pool on systems with big number of CPU cores. This slows down query processing significantly.	2021-07-06 18:26:56 +03:00
Aliaksandr Valialkin	b805a675f3	lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects This should reduce memory usage on systems with big number of CPU cores, since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains a separate pool inmemoryPart objects per each CPU core. Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized by big sizes of inmemoryPart objects.	2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin	d8e7c1ef27	lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method This reduces the load on memory allocator in Go runtime in production workload.	2021-07-06 16:33:24 +03:00
Aliaksandr Valialkin	db6bd69475	lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small. This results in the increased load on memory allocation code in Go runtime. Increase the pool capacity in order to reduce the load on Go runtime.	2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin	fd32855a6c	lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered (aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.	2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin	22c6e64bbc	lib/storage: consistency renaming: tagCache -> tagFiltersCache This improves code readability	2021-07-06 11:03:30 +03:00
Aliaksandr Valialkin	21abf487c3	lib/workingsetcache: properly update stats for requests and cache misses Previously the stats for cache misses could be improperly counted, because it had inflated cache misses if the entry was missing in the curr cache, but was existing in the prev cache. The same applies to cache requests - they were inflated if the entry was missing in the curr cache.	2021-07-06 10:54:38 +03:00
Roman Khavronenko	0541e3108a	add option to add Copy button for code snippets (#1433 ) To add a Copy button wrap code snippet with the following element: ``` <div class="with-copy" markdown="1"> <your-code-snippet> </div> ``` See the changes to `Kubernetes monitoring with VictoriaMetrics Single` for details.	2021-07-06 10:54:37 +03:00
Aliaksandr Valialkin	e5031d9aee	lib/workingsetcache: fix cache capacity calculations after `4f0003f182`	2021-07-05 17:16:35 +03:00
Aliaksandr Valialkin	bd71f102e8	lib/workingsetcache: typo fixes after `d0c830039d`	2021-07-05 15:35:51 +03:00
Roman Khavronenko	e2b5f93170	move github-pages docs to the main repo (#1432 ) * move github-pages docs to the main repo * rm github actions for copying docs to VictoriaMetrics/VictoriaMetrics.github.io	2021-07-05 15:20:29 +03:00
Aliaksandr Valialkin	4b25e627f8	lib/workingsetcache: properly switch to `whole` mode Previously the switch from `split` to `whole` mode had been performed too early, e.g. when the current cache size became bigger than 1/4 of the allowed cache size. Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size. This change can reduce memory usage for data ingestion path when big number of active time series are ingested.	2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin	51516b96e6	lib/storage: tune cache sizes according to production workload	2021-07-05 15:14:45 +03:00
Aliaksandr Valialkin	f12f97daa1	lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes One minute cache timeout result in slower queries in some production workloads where the interval between query execution is in the range 1 minute - 2 minutes.	2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin	377bb06b47	lib/cgroup: set GOGC to 50 by default if it isn't set This should reduce memory usage for typical VictoriaMetrics workloads by up to 50%	2021-07-05 12:34:01 +03:00
Aliaksandr Valialkin	28a778dc9f	docs/CHANGELOG.md: document the bugfix for `vm_merge_need_free_disk_space` metric at `9a83e9018d`	2021-07-05 12:01:24 +03:00
Aliaksandr Valialkin	2386829ad6	docs/Articles.md: add an url to https://medium.com/ibm-garage/monitoring-of-multiple-openshift-clusters-with-victoriametrics-d4f0979e2544	2021-07-05 11:52:18 +03:00
Aliaksandr Valialkin	8055439fe4	lib/storage: properly detect free disk space shortage during data merge Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin	44855f0c9b	app/{vmselect,vmstorage}: clarify the description for `-dedup.minScrapeInterval` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1426	2021-07-02 15:06:41 +03:00
Aliaksandr Valialkin	6fc3696260	lib/promscrape/discovery/consul: use case-insensitive comparison for service names Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424	2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin	61e483a01c	lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime This is a follow-up for `b84aea1e6e`	2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin	72de54f93e	lib/promauth: cache the client TLS certificate for up to a second This should reduce CPU usage when TLS connections are established at a high rate.	2021-07-02 13:20:18 +03:00
Aliaksandr Valialkin	a12c6a52c9	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.15 to v1.0.16 This should help with proper copying of tls.Config struct after it has been set up in lib/promauth. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2021-07-02 12:05:23 +03:00
Aliaksandr Valialkin	4b4e69791f	docs/Cluster-VictoriaMetrics.md: typo fix: siplify -> simplify	2021-07-02 10:50:26 +03:00
Aliaksandr Valialkin	9ec08213be	docs/Cluster-VictoriaMetrics.md: add a chapter describing a toy cluster setup on a single host While at it, refer to available tools, which can simplify cluster setup	2021-07-02 10:48:13 +03:00
Aliaksandr Valialkin	1c12c0f79c	lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2021-07-01 15:43:43 +03:00
Nikolay	6bd2309449	fixes /targets button style (#1423 ) * fixes /targets button style https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422 * updates boostrap version	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	c1b15c7764	docs/CHANGELOG.md: document `ae485c2bfd`	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	71c856beb8	lib/workingsetcache: reset the cache mode when the cache is reset This should reduce memory usage if the working set is reduced after the cache reset.	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	225ec527df	deployment/docker: upgrade alpine image from v3.13.5 to v3.14.0	2021-07-01 10:57:32 +03:00
Roman Khavronenko	c6cf821600	dashboard: several minor fixes (#1418 ) * move panel `Disk writes/reads` to `Resource usage` row * rename row `storage` to `vmstorage` * remove cumulative display for `Storage ETA` panel	2021-07-01 05:45:35 +03:00
Aliaksandr Valialkin	bced9ee666	lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute This should reduce memory usage on a system with high number of active time series and a high churn rate. One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).	2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin	b7c0b3dde3	lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs This should reduce memory usage for the pool on systems with big number of CPU cores. The sync.Pool maintains per-CPU pools, so the total number of objects in the pool is proportional to the number of available CPU cores. The channel limits the number of pooled objects by its own capacity. This means smaller number of pooled objects on average.	2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin	2edfea8c36	lib/promscrape/discovery/docker: fix golint warning: `struct field Id should be ID`	2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin	609ad6d9bf	lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache This should prevent from stats overwriting when the previous indexdb is queried.	2021-06-29 13:11:32 +03:00
Aliaksandr Valialkin	0c4c630839	lib/promscrape: typo fix in `/targets` output The typo has been introduced in `fb72a2133f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408	2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin	4d8ab5d9fa	docs/vmagent.md: mention about docker_sd_config support	2021-06-25 20:53:09 +03:00
Aliaksandr Valialkin	db66ea9a5f	docs/CHANGELOG.md: cut v1.62.0	2021-06-25 13:29:48 +03:00
Aliaksandr Valialkin	97d1ccfc8e	lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common This is a follow up after `c85a5b7fcb`	2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin	4461e20e7d	lib/promscrape: consistently sort service discovery routines This should simplify further maintenance of the code	2021-06-25 13:22:16 +03:00
Lu Jiajing	12b4cbb68f	Support Docker ServiceDiscovery (#1402 ) * add docker discovery * add test * add labels test and add scrape work * remove TODO * refactor to merge apiConfig and sdConfig * apply suggestion	2021-06-25 13:22:16 +03:00
Nikolay	501429c3ff	adds missing MustStop call to do and http sd (#1404 )	2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin	f1a2aa0992	vendor: `make vendor-update`	2021-06-24 17:28:37 +03:00
Aliaksandr Valialkin	01af676436	docs: consistently put the link to articles and slides about VictoriaMetrics after the links to case studies	2021-06-24 15:38:49 +03:00
Aliaksandr Valialkin	634f2128d8	docs/CaseStudies.md: add a case study for DFKI	2021-06-24 15:25:19 +03:00
Aliaksandr Valialkin	b054563703	Add case study for Groove X	2021-06-24 15:06:19 +03:00
Aliaksandr Valialkin	65576ebb5b	docs/CaseStudies.md: add Sensedia case study	2021-06-24 14:35:47 +03:00
Aliaksandr Valialkin	1b6850bab3	docs/CHANGELOG.md: document the bugfix in increase_pure() function from the commit `fb4f758715`	2021-06-24 12:06:15 +03:00
Aliaksandr Valialkin	b84aea1e6e	lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores	2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin	856aecae05	app/vmselect/promql: return the last timestamp for the max / min value from `tmax_over_time()` and `tmin_over_time()` function as most users expect	2021-06-23 14:18:37 +03:00
Aliaksandr Valialkin	dffa2afefe	docs/CHANGELOG.md: document the bugfix for incorrect stats collection for concurrently executed tag filter Follow up for `c22114c6f0`	2021-06-23 14:06:33 +03:00
Aliaksandr Valialkin	c18017a9c3	app/vminsert/netstorage: sort the `-storageNode` list passed to `vminsert` nodes This should reduce resource usage (CPU, RAM, disk IO) at vmstorage nodes if the addresses of vmstorage nodes are passed in random order to vminsert nodes.	2021-06-23 14:00:08 +03:00
Aliaksandr Valialkin	a22f37599b	lib/storage: tune tag filters search logic Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624 The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats. This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters for further queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin	f10fa0d1d7	lib/promscrape/discovery/consul: properly pass namespace to Consul watcher Follow-up for `58a2989fe7`	2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin	4adf6c9766	lib/promscrape/discovery/http: follow up after `e307bbb29a`	2021-06-22 13:42:10 +03:00
Nikolay	e03a3d3a36	adds http_sd (#1399 ) * adds http_sd * adds X-Prometheus-Refresh-Interval-Seconds header * Update lib/promscrape/discovery/http/api.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin	3ab3902f17	lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does	2021-06-22 13:18:51 +03:00
Aliaksandr Valialkin	61fc9b98e5	docs/CHANGELOG.md: document the support for Consul namepsace See `58a2989fe7`	2021-06-22 12:56:12 +03:00
Nikolay	827a2396d2	adds consul enterprise namespace support (#1400 ) * adds consul enterprise namespace support * Update lib/promscrape/discovery/consul/consul.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin	a575882ca2	docs/PerTenantStatistic.md: document that the per-tenant statistic is a part of cluster version of VictoriaMetrics	2021-06-22 12:43:30 +03:00
Roman Khavronenko	79474baf99	vmctl: add more context to flags description in vm-native mode (#1395 )	2021-06-18 19:20:52 +03:00
Aliaksandr Valialkin	23fcafd437	docs/CHANGELOG.md: typo fixes	2021-06-18 19:15:29 +03:00
Aliaksandr Valialkin	b92d110cad	app/vmselect: log slow requests to all the `/api/v1/*` handlers if their execution time exceeds `-search.logSlowQueryDuration`	2021-06-18 19:07:03 +03:00
Aliaksandr Valialkin	d29e130181	docs/Single-server-VictoriaMetrics.md: mention that it is recommended to use a single scrape_interval across all the scrape targets	2021-06-18 15:41:35 +03:00
Aliaksandr Valialkin	4acc4602b3	app/vmctl: limit JSON line size by 10K samples (#1394 ) This should reduce the maximum memory usage at VictoriaMetrics when importing time series with big number of samples.	2021-06-18 15:41:34 +03:00
Aliaksandr Valialkin	3ec3705943	docs/Cluster-VictoriaMetrics.md: clarify docs about VictoriaMetrics cluster architecture	2021-06-18 14:35:55 +03:00
Aliaksandr Valialkin	cd697b88c5	docs/CHANGELOG.md: document the reduced disk write IO usage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-18 14:03:42 +03:00
Aliaksandr Valialkin	f9069ba32a	lib/promscrape: show jobs with empty scrape targets on /targets page	2021-06-18 10:54:12 +03:00
Roman Khavronenko	5cb378f5b5	dasbhoard: display tweaks (#1387 ) * rm cumulative visualisation for panel `Disk space used`. It uses % threshold and cumulative display breaks it. * remove area filling for resource usage row; * add job name for panels in resource usage row.	2021-06-18 10:48:40 +03:00
Nikolay	9ea1dca3dd	fixes DO service discovery labels (#1389 ) adds test for digitalocean sd	2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin	60bc35f550	docs/{vmgateway,vmbackupmanager}: explicitly mention that these components are a part of an enterprise package	2021-06-17 17:19:13 +03:00
Aliaksandr Valialkin	a207be3ffb	lib/storage: fix infinite loop introduced in `aa9b56a046` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin	0efd37cec1	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit `56b6b893ce` . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin	51fc469642	app/vmagent/remotewrite: `go fmt` after `0a796f7c3a`	2021-06-17 13:51:40 +03:00
Aliaksandr Valialkin	90c3606269	docs/vmagent.md: sync with app/vmagent/README.md via `make docs-sync`	2021-06-16 12:37:55 +03:00
Aliaksandr Valialkin	644102b03b	docs/CHANGELOG.md: document the changed `-remoteWrite.queues` value This is a follow-up for `0a796f7c3a` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1385	2021-06-16 12:37:55 +03:00
Zongyang	cf506e300d	Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUs * 2 (#1385 ) * Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUS() * 2 to reduce scrape interval Default value of vmagent option '-remotewrite.queues' is 4 and default size of vmagent ScheudleUnmarshalWorkers is number of CPUs, when available CPUs is much greater than 4, e.g 32, worker are competing push queues which will increase scrape interval and may cause scrape timeout. * Update README and flag description Co-authored-by: xiaozy <xiaozy01@fenbi.com>	2021-06-16 12:37:55 +03:00
Roman Khavronenko	a15c947045	promql: fix `increase_pure` calculation for cases with stale series (#1381 ) Due to staleness handling, increase_pure were using incorrect previous value during calculation in cases where series disappears for period longer than staleness period and then returns back. The fix suppose to account for a real datapoint value before staleness takes place. The fix should remove unexpected spikes while using `increase_pure` for staled series.	2021-06-15 17:37:51 +03:00
Aliaksandr Valialkin	234152d66e	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics works great with APM workloads (aka Application Performance Monitoring)	2021-06-15 17:33:41 +03:00
Aliaksandr Valialkin	b133de1e37	lib/storage: move deletedMetricIDs set from indexDB to Storage This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB). This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation. See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 . Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart. This should be OK in most cases.	2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin	ebaf68bcb0	lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error This is a follow-up for `af90c3c43b`	2021-06-14 15:20:38 +03:00
faceair	2ea187e801	lib/protoparser: stop read when callback error (#1380 )	2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin	5f91a701fa	lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377	2021-06-14 14:04:35 +03:00
Aliaksandr Valialkin	fb8114ad9c	vendor: update github.com/klauspost/compress from v1.13.0 to v1.13.1	2021-06-14 13:42:05 +03:00
Roman Khavronenko	db39c4a7d1	dashboard: bump version requirements (#1379 )	2021-06-14 13:32:32 +03:00
Aliaksandr Valialkin	5cd50d840f	docs/CHANGELOG.md: document the addition of DigitalOcean service discovery Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367	2021-06-14 13:19:31 +03:00
Nikolay	e42da47608	adds digital ocean sd (#1376 ) * adds digital ocean sd config * adds digital ocean sd https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367 * typo fix	2021-06-14 13:19:29 +03:00
Roman Khavronenko	1053d3e5a9	Dashboard cluster (#1375 ) * dashboard: update vmagent dash The update contains the following changes: * display anonymous memory usage metric. This metric suppose to reflect memory usage of the process which can't be freed by OS; * add legends to all panels. This is important for cases when users share the screenshots; * modify panels for Grafana v8.0.0 * dashboard: update cluster dash The update contains the following changes: * move stats panels to Configuration row, so it can be collapsed; * display anonymous memory usage metric. This metric suppose to reflect memory usage of the process which can't be freed by OS; * add legends to all panels. This is important for cases when users share the screenshots; * modify panels for Grafana v8.0.0	2021-06-14 13:03:54 +03:00
Aliaksandr Valialkin	df057177a0	lib/promscrape: increase the duration for reading the full response in stream parsing mode Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin	074b11fa69	lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin	357dbe092c	docs/Cluster-VictoriaMetrics.md: add lists for command-line flags for cluster components	2021-06-14 12:21:22 +03:00
Aliaksandr Valialkin	87d221f78a	lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin	52efd5a05c	docs/vmalert.md: follow-up after `6d5a8c28cd`	2021-06-14 11:43:02 +03:00
Roman Khavronenko	c5f493db8e	Vmalert docs (#1372 ) * vmalert: mention what happens if `for` is set to 0 or omitted * vmalert: add more context to docs	2021-06-14 11:43:01 +03:00
Aliaksandr Valialkin	541429a9af	docs/CHANGELOG.md: cut v1.61.1	2021-06-11 13:02:04 +03:00
Aliaksandr Valialkin	0672cfffa2	app/vmauth: properly handle http.ErrAbortHandler panic This panic can be raised by the reverseProxy on aborted request to the backend. So handle it (e.g. suppress) at reverseProxy.ServeHTTP call. Do not suppress the panic at lib/httpserver generic HTTP handler, since it may result in an inconsistent state left after the panicking handler. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353	2021-06-11 12:54:37 +03:00
Aliaksandr Valialkin	ce10bdc82a	lib/storage: reset cache on disk during series deletion and during indexdb rotation This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347	2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin	d5973d3180	docs/CHANGELOG.md: document the bugfix from `7adfe878e1`	2021-06-11 11:30:07 +03:00
Roman Khavronenko	f3cb2158a3	vmalert: fix mistake with object reuse while parsing response (#1370 ) * vmalert: fix mistake with object reuse while parsing response During the refactoring, the wrong optimisations was applied in parse function which caused metric fields reset. The change removes optimisation. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1369 * vmalert: add test to cover multiple metrics in one response	2021-06-11 11:30:07 +03:00
John Belmonte	3e79f3994e	spelling fix: synonym (#1363 )	2021-06-11 10:58:48 +03:00
Aliaksandr Valialkin	eb335d2c29	lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard	2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin	d06c0e7a94	lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores	2021-06-11 10:49:02 +03:00
Aliaksandr Valialkin	56192f24b4	vendor: `make vendor-update`	2021-06-09 20:41:58 +03:00
Aliaksandr Valialkin	83adc2f3ac	docs/CHANGELOG.md: cut v1.61.0	2021-06-09 19:04:59 +03:00
Aliaksandr Valialkin	c7efd5b43f	docs/FAQ.md: add a chapter comparing VictoriaMetrics to QuestDB	2021-06-09 19:03:47 +03:00
Aliaksandr Valialkin	e8e7f03394	app/vmselect/promql: typo fix in the comment	2021-06-09 18:34:57 +03:00
Aliaksandr Valialkin	203d883b2b	docs/Articles.md: update the broken link to https://nordicapis.com/api-monitoring-with-prometheus-grafana-alertmanager-and-victoriametrics/	2021-06-09 16:40:39 +03:00
Aliaksandr Valialkin	247b2a5a08	app/vmauth: improve readability for a config with multiple `src_paths`	2021-06-09 15:38:09 +03:00
Aliaksandr Valialkin	1e13deaa2c	docs/CHANGELOG.md: document the enterprise bugfix for the `target` property in Graphite Render API	2021-06-09 13:51:32 +03:00
Aliaksandr Valialkin	490783696a	docs/CHANGELOG.md: document improvements in re-routing handling in vminsert See the following commits: * `1c09e71f5b` * `0d067eb112` * `2c6b917749`	2021-06-09 13:42:12 +03:00
Aliaksandr Valialkin	520d62ade2	docs/vmagent.md: mention that vmagent supports scrape targets sharding	2021-06-09 12:30:54 +03:00
Aliaksandr Valialkin	f3749dedba	docs: document rules replay feature for vmalert Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836 This is a follow-up for `2a259ef5e7`	2021-06-09 12:30:54 +03:00
Roman Khavronenko	5aa7846900	vmalert: support rules backfilling (aka `replay`) (#1358 ) * vmalert: support rules backfilling (aka `replay`) vmalert can `replay` configured rules in the past and backfill results via remote write protocol. It supports MetricsQL/PromQL storage as data source, and can backfill data to remote write compatible storage. Supports recording and alerting rules `replay`. See more details in README. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836 * vmalert: review fixes * vmalert: readme fixes	2021-06-09 12:30:54 +03:00
Aliaksandr Valialkin	8ad445474a	docs/CHANGELOG.md: document the bugfix, which prevents panics for aborted http requests in vmauth See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353 This is a follow-up for `6b29b955c0`	2021-06-09 12:12:47 +03:00
Nikolay	2c1611d316	disables panic for net/httpAbortHandler (#1355 )	2021-06-09 12:12:45 +03:00
k1rk	c6c789db8f	rename serviceHealth group name to vm-health (#1360 ) this causes conflicts in `victoria-metrics-k8s-stack` chart =)	2021-06-09 02:26:21 +03:00
Aliaksandr Valialkin	8890dadd73	deployment/docker/docker-compose.yml: update Grafana from v7.5.2 to v8.0.0 See https://github.com/grafana/grafana/releases/tag/v8.0.0	2021-06-08 15:53:24 +03:00
Aliaksandr Valialkin	2f8baf2aa7	vendor: `make vendor-update`	2021-06-08 15:53:23 +03:00
Aliaksandr Valialkin	661f6f929b	deployment/docker: update Go builder from v1.16.4 to v1.16.5 See the fixed isses at https://github.com/golang/go/issues?q=milestone%3AGo1.16.5+label%3ACherryPickApproved	2021-06-08 15:44:00 +03:00
Aliaksandr Valialkin	f47d57a6c2	vendor: update github.com/klauspost/compress from v1.12.3 to v1.13.0	2021-06-08 15:42:57 +03:00
Aliaksandr Valialkin	1e4a64844d	lib/storage: properly account the number of loops spent when matching for `or suffixes` This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin	2c6b917749	app/vminsert/netstorage: update storageNode.lastRerouteTime before the rerouting This is needed for reliable detection of storage nodes with recent rerouting	2021-06-08 12:06:32 +03:00
Aliaksandr Valialkin	0d067eb112	app/vminsert/netstorage: tune re-routing algorithm Do not re-route data to unavailable storage node. Send it to the remaining storage nodes instead even if they cannot keep up with the load. This should spread the load more evenly among available storage nodes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165	2021-06-05 16:23:44 +03:00
Aliaksandr Valialkin	e7d353ee6a	lib/promrelabel: add tests for labelsToString() function	2021-06-04 20:42:14 +03:00
Aliaksandr Valialkin	269e35d676	app/{vmagent,vminsert}: follow-up after `2fe045e2a4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343	2021-06-04 20:33:22 +03:00
jelmd	d8b46908db	new feature: debug relabeling (#1344 ) * new feature: relabel logging Use scrape_configs[x].relabel_debug = true to log metric names inkl. labels before and after relabeling. After relabeling related metrics get dropped, i.e. not submitted to servers. * vminsert wants relabel logging, too.	2021-06-04 20:33:21 +03:00
Nikolay	3d89c01d07	fixes solaris build (#1345 )	2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin	67cfc07004	docs/CHANGELOG.md: document the bugfix from `6f19bb23a1`	2021-06-04 11:56:06 +03:00
Hason Chan	439c2ed510	fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350 )	2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin	c53a90e5fc	docs/CHANGELOG.md: document that it is possible to build VictoriaMetrics components for Solaris Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322 This is a follow-up for `ddc8022702`	2021-06-04 04:41:12 +03:00
Aliaksandr Valialkin	1c09e71f5b	app/vminsert: add `-disableRerouting` command-line flag for disabling re-routing if some vmstorage nodes have lower performance than the others Refactor the rerouting mechanism and make it more resilient to cases when some of vmstorage nodes are temporarily unavailable. Reduce the probability of rerouting storm. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165	2021-06-04 04:33:52 +03:00
Aliaksandr Valialkin	fc2565b4ee	lib/storage: reduce memory allocations when syncing dateMetricIDCache	2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin	8d2553e6a1	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.8 to v1.6.0	2021-05-31 09:15:01 +03:00
Aliaksandr Valialkin	8cdecfc52c	app/vmauth: allow balancing the load among multiple backend nodes by specifying multiple urls in `url_prefix` config	2021-05-29 01:04:22 +03:00
Aliaksandr Valialkin	bf9f77d74d	docs/Articles.md: add a link to https://www.percona.com/blog/2021/05/26/compiling-a-percona-monitoring-and-management-v2-client-in-arm-raspberry-pi-3/	2021-05-28 14:34:06 +03:00
Aliaksandr Valialkin	b782fb8ab8	vendor: `make vendor-update`	2021-05-28 13:16:37 +03:00
Aliaksandr Valialkin	c97212ea63	docs/MetricsQL.md: add a link to technical details about `rate()` and `increase()` calculations in Prometheus and VictoriaMetrics	2021-05-28 13:14:23 +03:00
Aliaksandr Valialkin	734074e8a6	docs/Single-server-VictoriaMetrics.md: remove misleading wording about querying Graphite metrics with MetricsQL	2021-05-28 02:40:02 +03:00
Aliaksandr Valialkin	0b9f0de0a1	lib/promscrape: fix tests after `f0c21b6300`	2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin	6865f3b497	Revert "lib/mergeset: remove a pool for inmemoryBlock structs" This reverts commit `793fe39921`. Reason to revert: production testing revealed possible slowdown when registering big number of new time series	2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin	7b33bc67a1	lib/mergeset: remove a pool for inmemoryBlock structs The pool for inmemoryBlock struct doesn't give any performance gains in production workloads, while it may result in excess memory usage for inmemoryBlock structs inside the pool during background merge of indexdb.	2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin	97de72054e	docs: document `f0c21b6300`	2021-05-27 15:04:13 +03:00
faceair	b801b299f0	lib/promscrape: apply body size & sample limit to stream parse (#1331 ) * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 15:04:11 +03:00
Aliaksandr Valialkin	49490ae5a7	lib/protoparser/clusternative: remove duplicate `cannot read packet size` phrase from the log message Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	c85084b659	lib/handshake: pass io.EOF unmodified to the caller for BufferedConn.Read, so it could properly detect the end of stream	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	10b2855949	lib/storage: fix spelling typo: `borken->broken` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	6c4921b3bd	docs: `make docs-sync` after `2bbb1cc7c1`	2021-05-26 12:33:01 +03:00
Roman Khavronenko	b42f7fc185	Docs review (#1330 ) * re-order components by prioritizing Cluster-VictoriaMetrics.md * drop Home.md since it just duplicates other links	2021-05-26 12:32:59 +03:00
Aliaksandr Valialkin	820ac6cd0c	docs/CHANGELOG.md: document changes from `2233d6ed8a` and `d210958fd0`	2021-05-26 12:24:28 +03:00
Roman Khavronenko	e183a5c532	vmalert: automatically reload configuration on file change (#1326 ) New flag `-rule.configCheckInterval` defines how often `vmalert` will re-read config file. If it detects any changes, the config will be reloaded. This behaviour is turned off by default. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/512	2021-05-26 12:24:27 +03:00
Aliaksandr Valialkin	6b90570ed3	lib/uint64set: store pointers to bucket16 instead of bucket16 objects in bucket32 This speeds up bucket32.addBucketAtPos() when bucket32.buckets contains big number of items, since the copying of bucket16 pointers is much faster than the copying of bucket16 objects. This is a cpu profile for copying bucket16 objects: 10ms 13.43s (flat, cum) 32.01% of Total 10ms 120ms 650: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 651: b.b16his[pos] = hi . 13.31s 652: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . . 653: b16 := &b.buckets[pos] . . 654: *b16 = bucket16{} . . 655: return b16 . . 656:} This is a cpu profile for copying pointers to bucket16: 10ms 1.14s (flat, cum) 2.19% of Total . 100ms 647: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 648: b.b16his[pos] = hi 10ms 700ms 649: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . 330ms 650: b16 := &bucket16{} . . 651: b.buckets[pos] = b16 . . 652: return b16 . . 653:}	2021-05-25 14:27:52 +03:00
Dan Fredell	e05c03cf00	Fix quote difference on label_move example (#1321 ) Fix quote difference on label_move example	2021-05-25 12:13:39 +03:00
Aliaksandr Valialkin	a0b001bfec	app/vmselect/netstorage: remove duplicate limiter on concurrent queries It duplicates the `-search.maxConcurrentRequests` limiter.	2021-05-24 19:13:04 +03:00
Aliaksandr Valialkin	25ed1f0c4f	docs/CHANGELOG.md: cut v1.60.0	2021-05-24 15:55:31 +03:00
Aliaksandr Valialkin	65b4ae95e3	docs/Single-server-VictoriaMetrics.md: clarify that the storage size depends on the number of samples per series	2021-05-24 15:48:45 +03:00
Aliaksandr Valialkin	c9229e3c0b	docs/vmalert.md: sync with app/vmalert/README.md via `make docs-sync`	2021-05-24 15:48:44 +03:00
Aliaksandr Valialkin	890e1bd826	app/vmagent/remotewrite: use WARN level instead of ERROR level for `couldnt send a block with size ... bytes to ...` log message This is really warning, since vmagent re-tries sending the data block until success.	2021-05-24 15:43:32 +03:00
Aliaksandr Valialkin	1c16cbacf5	lib/storage: do not stop data ingestion on the first error in Storage.AddRows Continue data ingestion for the rest of blocks.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	2601844de3	lib/storage: limit the number of rows per each block in Storage.AddRows() This should reduce memory usage when ingesting big blocks or rows.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	95b735a883	lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows This should reduce memory usage a bit on data ingestion path	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	0f84503880	lib/bloomfilter: fix TestLimiterConcurrent	2021-05-24 05:18:29 +03:00
Aliaksandr Valialkin	1487274209	vendor: update github.com/valyala/gozstd from v1.10.0 to v1.11.0	2021-05-24 05:00:53 +03:00
Aliaksandr Valialkin	745eda9e87	lib/fs: do not pass done callback to tryRemoveAll() func This improves code readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-24 05:00:53 +03:00
Aliaksandr Valialkin	402a8ca710	lib/storage: do not populate MetricID->MetricName cache during data ingestion This cache isn't needed during data ingestion, so there is no need in spending RAM on it. This reduces RAM usage on data ingestion path by 30%	2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin	68c3901ebd	docs/CHANGELOG.md: small typo fix	2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin	8ec3e876be	docs/CHANGELOG.md: document the addition of `extra_filter_labels` at `84cc0513e1`	2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin	0fc857d363	lib/{mergeset,storage}: reduce the number of IFNO log messages like `merged ... items across ... blocks in ... seconds` Log these messages if the merge takes more than 30 seconds instead of 10 seconds.	2021-05-23 14:15:49 +03:00
Roman Khavronenko	beee24ecee	vmalert: support `extra_filter_labels` setting per-group (#1319 ) The new setting `extra_filter_labels` may be assigned to group. If it is, then all rules within a group will automatically filter for configured labels. The feature is well-described here https://docs.victoriametrics.com#prometheus-querying-api-enhancements New setting is compatible only with VM datasource.	2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin	71ff7ee18d	lib/promauth: follow-up after `5b8176c68e`	2021-05-22 18:02:03 +03:00
Nikolay	2780d6dbcd	basic OAuth2 support for remoteWrite and scrape targets (#1316 ) * adds OAuth2 support for remoteWrite and scrapping * adds tests changes init	2021-05-22 18:02:01 +03:00
Aliaksandr Valialkin	89e1a45cdb	lib/fs: concurrently remove up to 1024 blocked NFS directories Previously the blocked directories were removed sequentially by a single goroutine. This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second, when big number of LSM parts are created and removed at high rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:58:08 +03:00
Aliaksandr Valialkin	23355ca34c	lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full This should reduce the probability of the panic on a highly loaded VictoriaMetrics accepting millions of samples per second. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:21:35 +03:00
Aliaksandr Valialkin	e9a63a5942	docs/MetricsQL.md: add a link to a list of supported timezones that can be passed to timezone_offset() function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306	2021-05-21 16:56:07 +03:00
Nikolay	23a6c9c016	changes vmalert query function (#1307 ) * changes vmalert query function for prometheus rules compatibility its better to use labels as map. it simplifies template evaluation and allow to ignore can't evaluate field error because map will return default value. fixes https://github.com/VictoriaMetrics/operator/issues/243	2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin	f664f7fb1d	docs/CHANGELOG.md: mention the bugfix from `d626c5c2a9` Updates https://github.com/VictoriaMetrics/operator/issues/243	2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin	d77db9d813	all: do not skip SIGHUP signal during service initialization This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin	45849886a3	docs/FAQ.md: re-order questions to be more attractive to visitors	2021-05-20 19:50:57 +03:00
Aliaksandr Valialkin	6139f6ed6d	app/vmauth: add ability to protect `/-/reload` endpoint with authKey	2021-05-20 18:48:34 +03:00
Aliaksandr Valialkin	b5f22f58cd	vendor: `make vendor-update`	2021-05-20 18:38:28 +03:00
Aliaksandr Valialkin	69e365cd48	Makefile: update golangci-lint from v1.29.0 to v1.40.1	2021-05-20 18:30:24 +03:00
Aliaksandr Valialkin	9aa22cccf0	docs/CHANGELOG.md: move tip to proper place	2021-05-20 17:58:24 +03:00
Aliaksandr Valialkin	4a7aef4707	docs/FAQ.md: add a question on how to run VictoriaMetrics on FreeBSD The question has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1284	2021-05-20 16:18:36 +03:00
Aliaksandr Valialkin	429ae37863	docs/FAQ.md: add `can I use VictoriaMetrics instead of Prometheus?` The question has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1284	2021-05-20 16:18:36 +03:00
Aliaksandr Valialkin	6d987971e3	docs/FAQ.md: add a question about memory limits for VictoriaMetrics components The question has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1284	2021-05-20 16:18:36 +03:00
Aliaksandr Valialkin	26dc52cbde	docs/FAQ.md: add a question about multi-tenancy The question has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1284	2021-05-20 15:52:17 +03:00
Aliaksandr Valialkin	da0b32c31a	app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour These numbers are exposed via the following metrics: - vmagent_hourly_series_limit_current_series - vmagent_daily_series_limit_current_series Expose also the limits via the following metrics: - vmagent_hourly_series_limit_max_series - vmagent_daily_series_limit_max_series	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	165a9f9200	app/vmstorage: add ability to limit series cardinality via `-storage.maxHourlySeries` and `-storage.maxDailySeries` command-line flags	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	7aad5c3f76	app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	a613be1518	docs/CHANGELOG.md: document the bugfix in vmctl import for InfluxDB lines with identical names for field and tag See `dcf8803bbd` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1299	2021-05-20 12:07:15 +03:00
Roman Khavronenko	12d0c6b6e0	vmctl: explicitly set `::tag` type for labels selector in `influx` mode (#1310 ) The `::tag` type is needed in cases when field and tag names are equal, which results into unexpected results in InfluxQL. Setting the type explicitly helps InfluxDB to understand which exact column we apply filter to. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1299	2021-05-20 12:07:15 +03:00
Aliaksandr Valialkin	2cc5567ab8	docs/CHANGELOG.md: refer to the issue related to timezone_offset() function	2021-05-20 12:02:32 +03:00
Aliaksandr Valialkin	110a888e39	lib/promscrape/discovery/kubernetes: make `golangci-lint` happy by removing empty branches	2021-05-20 12:00:17 +03:00
Aliaksandr Valialkin	180829b8c2	app/vmselect/promql: add `timezone_offset(tz)` function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306	2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin	e228f479a5	lib/storage: remove possible data race when logging dropped labels	2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin	dcac849c1f	app/vmagent/remotewrite: sort labels before sending the series to per-remoteWrite.url queues	2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin	9d97f44772	lib/promscrape/discovery/kubernetes: reload objects on object parse error Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 23:27:24 +03:00
Neo He	c5ab00ebee	app/{vmbackup,vmrestore},docs/vmrestore.md: typo fix: vbackup -> vmbackup (#1305 )	2021-05-18 16:38:15 +03:00
Aliaksandr Valialkin	74ef40034c	lib/httpserver: typo fix in `-http.shutdownDelay` command-line flag description: servier -> server	2021-05-18 16:25:27 +03:00
Aliaksandr Valialkin	1668280e67	docs/vmalert.md: document multitenant support https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740	2021-05-18 16:25:21 +03:00
Aliaksandr Valialkin	c507faec0b	lib/promscrape/discovery/kubernetes: simplify the reload logic for urlWatcher.objectsByKey	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	0f54c0121b	lib/promscrape/discovery/kubernetes: properly update vm_promscrape_discovery_kubernetes_scrape_works metric Previously it wasn't descreased during config update.	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	9f62d348db	lib/promscrape/discovery/kubernetes: log errors and stop service discovery when unexpected updates are received from Kubernetes API server Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	7fe362deb1	app/vmauth: reload `-auth.config` on the request to `/-/reload` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194	2021-05-18 02:24:37 +03:00
Aliaksandr Valialkin	25ca108642	docs/vmbackup.md: typo fix: snaphosts -> snapshots Thanks to @jelmd - see `1ab27582a3 (r50884395)`	2021-05-18 01:14:01 +03:00
Aliaksandr Valialkin	6ea191d196	docs: dealay -> delay	2021-05-18 01:07:32 +03:00
Aliaksandr Valialkin	c4ed50ae54	lib/promrelabel: add tests for conditional removal of label on another label match Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1294	2021-05-18 00:23:23 +03:00
Aliaksandr Valialkin	8764b0ae21	lib/promscrape/discovery/kubernetes: key ScrapeWork objects by urlWatcher instead of namespace This makes the code less fragile if urlWatcher would depend on additional to namepsace properties. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-17 23:49:48 +03:00
Aliaksandr Valialkin	bae4d61ef2	docs/CHANGELOG.md: document `b38edec7ee` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1293	2021-05-17 01:58:18 +03:00
Aliaksandr Valialkin	21340ebd4f	vendor: `make vendor-update`	2021-05-17 01:51:55 +03:00
Roman Khavronenko	c47825c255	Docs update from victoriaMetrics.github.io (#1302 ) * port change from `11ca65677b` * port change from `afb41dfa43` * port change from `f82e3733c9` * port change from `d499ab0502`	2021-05-17 01:51:54 +03:00
Roman Khavronenko	3428df6f15	vmalert: use stringified label keys for duplicates map in recroding rules (#1301 ) duplicates map helps to determine wheter extra labels has overriden labels which make time series unique. It was using a sorted hashed labels sequence as a key. But hashing algorithm could have collisions, so it is more convenient to not use hashing at all. Log message for recording rules duplicates was improved as well. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1293	2021-05-17 01:51:48 +03:00
Aliaksandr Valialkin	9c8a411e08	vendor: update github.com/valyala/gozstd from v1.9.0 to v1.10.0	2021-05-17 01:41:51 +03:00
Aliaksandr Valialkin	e08287f017	lib/promscrape: reload auth tokens from files every second Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:03:35 +03:00
Aliaksandr Valialkin	f13fb6e867	docs/Articles.md: add a link to https://fly.io/blog/measuring-fly/	2021-05-14 20:03:34 +03:00
Aliaksandr Valialkin	30e8ba63f1	docs/vmauth.md: sync with app/vmauth/README.md after `10a47af631`	2021-05-14 18:13:35 +03:00
Aliaksandr Valialkin	a6cb4f10a7	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin	d274dae73f	docs/Single-server-VictoriaMetrics.md: document how to reduce memory usage when importing too long JSON lines into VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1295	2021-05-14 17:22:51 +03:00
Denys Holius	ac5a28db9a	Fix Cortex typo	2021-05-13 17:40:19 +03:00
Aliaksandr Valialkin	23afbd5094	app/vmagent/remotewrite: clarify the comment explaining why vmagent drops blocks if remote storage returns 400 or 409 status code	2021-05-13 16:17:09 +03:00
Aliaksandr Valialkin	e3f61d540b	lib/promscrape: limit `scrape_timeout` by `scrape_interval` like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1281	2021-05-13 16:10:42 +03:00
Aliaksandr Valialkin	76e03b46df	docs/CHANGELOG.md: document the bugfix from `b4f5be8bd8`	2021-05-13 11:19:31 +03:00
匠心零度	d5285ecaf0	fix vagent imbalance problem (#1292 ) /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-05-13 11:19:30 +03:00
Aliaksandr Valialkin	f13585dc5d	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:09 +03:00
Aliaksandr Valialkin	d13906bf1f	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:07 +03:00
Aliaksandr Valialkin	66c6976723	lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for `b50024812e`	2021-05-13 09:27:35 +03:00
Nikolay	8743bf541f	adds cgroupsv2 support (#1283 ) * adds cgroupv2 limits support https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269 * small fix * changes Atoi to ParseUint	2021-05-13 09:27:33 +03:00
Aliaksandr Valialkin	2839055513	lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss	2021-05-13 09:01:05 +03:00
Aliaksandr Valialkin	008ae25b3a	lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate These functions are non-trivial, while their code has minimal differences. It is better from maintainability PoV to merge these functions into a single function.	2021-05-12 18:01:08 +03:00
Aliaksandr Valialkin	ad04c29a35	docs/Single-server-VictoriaMetrics.md: typo fix: retuns->returns	2021-05-12 17:23:27 +03:00
Nikolay	be87be34a4	Adds tsdb match filters (#1282 ) * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin	56b08390f6	app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode `vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.	2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin	027607db3e	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices scrape targets every 5 seconds, since they may depend on changed service and pod objects This should make endpoints and endpointslices scrape targets eventually consistent with the maximum delay of 5 seconds after the related service or pod object changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-12 14:12:43 +03:00
Aliaksandr Valialkin	1d32b008c6	lib/httpserver: add new X-Server-Hostname header instead of overwriting already exsiting header This makes possible tracking origins of chained requests over multiple hops.	2021-05-11 23:47:19 +03:00
Aliaksandr Valialkin	f1317f7c6c	lib/httpserver: return X-Server-Hostname http header in all the responses for better debuggability	2021-05-11 22:04:41 +03:00
Aliaksandr Valialkin	4e59cf4380	lib/storage: properly apply time range when matching an empty filter It must match all the time series on the given time range. Previously it was matched to all the time series without the restriction on the given time range.	2021-05-11 01:09:35 +03:00
Aliaksandr Valialkin	dea1cdd817	Revert ".github/dependabot.yml: remove automated dependency version checks" This reverts commit `5b986c95dd`. This check verifies only dependencies needed for github-actions. This is OK.	2021-05-10 12:06:19 +03:00
Aliaksandr Valialkin	36a79b7fd3	Add `make check-licenses` rule for the ability to manually check licenses in vendored dependencies This is a follow-up for `c687536956`	2021-05-10 12:01:37 +03:00
Aliaksandr Valialkin	15ad9deccf	.github/dependabot.yml: remove automated dependency version checks Dependency updates must be under manual control, since the resulting code diffs must be reviewed manually for the sake of security. It is done with `make vendor-update` now.	2021-05-10 12:01:29 +03:00
Artem Navoiev	632eb82dbd	Add vendor license checker, update codecov action, add dependbot for … (#1280 ) * Add vendor license checker, update codecov action, add dependbot for github actions * update gitingore, temprorary turn on check * fix action name * change action rules to trigger only when vendor changes * remove obsolete line from main action	2021-05-10 12:01:20 +03:00
Aliaksandr Valialkin	cca9670573	docs/CHANGELOG.md: document `-datasource.roundDigits` added at `5c448126dc`	2021-05-10 11:18:58 +03:00
Roman Khavronenko	a7f00101f5	vmalert: add support for `round_digits` param in datasource package (#1278 ) Starting from v1.56.0 VM supports `round_digits` which allows to limit the number of digits after the decimal point in response value. The feature can be used to reduce entropy of produced by recording rules values and significantly improve the compression. See more details in link below. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525	2021-05-10 11:18:56 +03:00
Aliaksandr Valialkin	f551b99082	docs/CHANGELOG.md: document vmalert fix for state restoration on startup	2021-05-10 11:10:34 +03:00
Roman Khavronenko	35237fe1f5	vmalert: fix error when rule didn't start if restore failed (#1279 ) Previously, `startGroup` could exit on restore errors despite the `remoteRead.ignoreRestoreErrors` flag value. Now vmalert checks the flag value before deciding whether to return error or just log it.	2021-05-10 11:10:32 +03:00
Aliaksandr Valialkin	2dddd68feb	docs/vmagent.md: add `stream parsing mode` chapter	2021-05-08 23:14:47 +03:00
Aliaksandr Valialkin	4128c4db16	docs/CHANGELOG.md: mention the comment, which gives an example of multi-level vminsert setup	2021-05-08 22:50:40 +03:00
Aliaksandr Valialkin	446577767f	vendor: `make vendor-update`	2021-05-08 22:50:37 +03:00
Aliaksandr Valialkin	326cf83eb4	lib/storage: remove dead code after the commit `3ccf7ea20c`	2021-05-08 20:15:59 +03:00
Aliaksandr Valialkin	9134ed93ab	deployment/dm: update Go builder from v1.16.3 to v1.16.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.16.4+label%3ACherryPickApproved for details	2021-05-08 20:03:34 +03:00
Aliaksandr Valialkin	9c505d27dd	lib/ingestserver: properly close incoming connections during graceful shutdown	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	4a5f45c77e	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	07bc021f58	app/vmalert: add missing comment for ErrStateRestore	2021-05-08 19:53:45 +03:00
Roman Khavronenko	78c388b246	dashboard: update descriptions for panel (#1275 ) This commit fixes panels descriptions for `Concurrent flushes on disk` (vmstorage) and `Concurrent inserts` (vminsert).	2021-05-07 11:25:00 +03:00
Aliaksandr Valialkin	c656b589a1	docs/Single-server-VictoriaMetrics.md: add links to vmauth and vmgateway as auth proxy examples	2021-05-07 10:46:47 +03:00
Aliaksandr Valialkin	e8478e1e97	app/vmbackup: make sure that `-snapshotName` isnt set if `-snapshot.createURL` is set	2021-05-07 08:44:44 +03:00
Aliaksandr Valialkin	3108cdb930	docs/CHANGELOG.md: document `904bbffc7f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-05 20:34:00 +03:00
Aliaksandr Valialkin	326c7995c1	docs/CHANGELOG.md: document `9cdd4696fe` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252	2021-05-05 20:33:59 +03:00
Roman Khavronenko	bb7e113dd4	vmalert: add flag to control behaviour on startup for state restore errors (#1265 ) Alerting rules now can return specific error type ErrStateRestore to indicate whether restore state procedure failed. Such errors were returned and logged before as well. But now user can specify whether to just log these errors (remoteRead.ignoreRestoreErrors=true) or to stop the process (remoteRead.ignoreRestoreErrors=false). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252	2021-05-05 12:24:32 +03:00
Aliaksandr Valialkin	e6c19cb09d	lib/promscrape/discovery/kubernetes: start watchers for pods and services before starting watchers for endpoints This should eliminate possible race when an update on endpoints depends on pods and/or services, which are missing in the cache yet. This could result in missing targets based on endpoints or endpointslices. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-05 12:23:16 +03:00
Aliaksandr Valialkin	43c52ff77a	lib/storage: use WARNING instead of INFO level for logging dropped labels	2021-05-03 13:57:28 +03:00
Aliaksandr Valialkin	ec6becd3f5	lib/httpserver: stop the process on panics in request handlers Panics may leave the process in inconsistent state. That's why it is better to stop the process after the panic instead of recovering from the panic. Unfortunately, the standard net/http.Server recovers panics in request handlers. See https://github.com/golang/go/issues/16542 . That's lib/httpserver must stop the process on itself after the panic.	2021-05-03 12:00:44 +03:00
Aliaksandr Valialkin	a302f79e5e	docs/CHANGELOG.md: document the bugfix for proper removal of stale parts (`477369b62f`)	2021-05-03 11:38:38 +03:00
Nikolay	62d58324dd	adds stalePartsRemover (#1261 ) for new created partitions	2021-05-03 11:34:33 +03:00
Aliaksandr Valialkin	60ffbcbb99	lib/promrelabel: add tests for removing the specified {label="value"} pair	2021-05-03 11:26:58 +03:00
Aliaksandr Valialkin	0d6cac112a	deployment/docker: update base docker image from alpine:3.13.2 to alpine:3.13.5	2021-05-01 11:04:59 +03:00
Aliaksandr Valialkin	9aa44a2760	docs/CHANGELOG.md: cut v1.59.0	2021-05-01 09:43:35 +03:00
Aliaksandr Valialkin	5f8181f7a1	vendor: `make vendor-update`	2021-05-01 09:43:33 +03:00
Aliaksandr Valialkin	b43ba6d85f	lib/storage: log dropped labels if the number of labels in a metric exceeds `-maxLabelsPerTimeseries` command-line flag value This should improve debuggability for this case.	2021-05-01 09:29:56 +03:00
Aliaksandr Valialkin	0a2e746175	docs/vmalert.md: update docs after `afca7b430c`	2021-04-30 11:49:40 +03:00
Roman Khavronenko	7394967841	vmalert: fix the typo in ApplyParams func (#1259 )	2021-04-30 11:47:11 +03:00
Aliaksandr Valialkin	131e9912eb	docs/Cluster-VictoriaMetrics.md: document `api/v1/series/count` endpoint	2021-04-30 11:41:45 +03:00
Roman Khavronenko	6fbedd62b8	vmalert: use rule's `evaluationInterval` as `step` param by default (#1258 ) User still can override param by specifying `datasource.queryStep` flag.	2021-04-30 10:03:50 +03:00
Aliaksandr Valialkin	daf2778025	docs/CHANGELOG.md: document the change from `f3a048288e` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232	2021-04-30 09:56:47 +03:00
Roman Khavronenko	b55677e93d	Vmalert: adjust `time` param for datasource queries according to `evaluationInterval` (#1257 ) * Simplify arguments list for fn `queryDataSource` to improve readbility * vmalert: adjust `time` param according to rule evaluation interval With this change, vmalert will start to use rule's evaluation interval for truncating the `time` param. This is mostly needed to produce consistent time series with timestamps unaffected by vmalert start time. Now, timestamp becomes predictable. Additionally, adjustment is similar to what Grafana does for plotting range graphs. Hence, recording rule series and recording rule expression plotted in grafana suppose to become similar in most of cases.	2021-04-30 09:56:46 +03:00
Aliaksandr Valialkin	8be1cb297b	app/vmagent: list user-visible endpoints at `http://vmagent:8429/` While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics	2021-04-30 09:38:23 +03:00
Nikolay	2eb8ef7b2b	changes vmalert Querier with per rule querier (#1249 ) * changes vmalert Querier with per rule querier it allows to changes some parametrs based on rule setting for instance - alert type, tenant for cluster version or event endpoint url.	2021-04-29 11:31:07 +03:00
Aliaksandr Valialkin	421a92983a	lib/promscrape/discovery/kubernetes: remove a mutex at urlWatcher - use groupWatcher mutex for accessing all the urlWatcher children This simplifies the code a bit and reduces the probability of improper mutex handling and deadlocks.	2021-04-29 10:17:45 +03:00
Nikolay	535b3ff618	vmagent kubernetes_sd tests (#1253 ) * first part of tests for kubernetes sd * makes linter happy * added more test cases * adds pub/sub for tests	2021-04-29 10:17:45 +03:00
Aliaksandr Valialkin	c434bb551e	vendor: update github.com/klauspost/compress from v1.12.1 to v1.12.2	2021-04-29 10:17:45 +03:00
Aliaksandr Valialkin	e37e1b1e34	lib/{storage,mergeset}: fix `unaligned 64-bit atomic operation` panic for 32-bit architectures The panic has been introduced in `56b6b893ce`	2021-04-27 16:42:19 +03:00
Aliaksandr Valialkin	2d1d60118d	lib/mergeset: split rows ingestion among multiple shards This improves rows ingestion on systems with many CPU cores by reducing lock contention. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244 Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243	2021-04-27 15:45:11 +03:00
Aliaksandr Valialkin	b3da457629	lib/promscrape/discovery/kubernetes: fix a deadlock introduced in `eddba29664` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 Thanks to @f41gh7 for providing the initial idea for deadlock fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1248	2021-04-27 14:59:56 +03:00
Aliaksandr Valialkin	cba2d13456	lib/storage: typo fix in info message when deleting the part outside the configured retention Previously the message was displaying incorrect retention time	2021-04-27 13:33:36 +03:00
Aliaksandr Valialkin	f14412321b	lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value	2021-04-27 00:26:32 +03:00
Roman Khavronenko	0ceb4f7565	vmalert: keep the returned timestamp when persisting recording rule (#1245 ) Previously, vmalert used `lastExecTime` timestamp when writing recording rules to the remote storage. This may be incorrect, if vmalert uses `datasource.lookback` flag, which means rule's expression will be executed at some moment in the past. To avoid such situations, vmalert now will use returned timestamp instead of `lastExecTime`.	2021-04-27 00:16:45 +03:00
Roman Khavronenko	2357e21024	docs: update per tenant stats page (#1246 )	2021-04-27 00:16:44 +03:00
Aliaksandr Valialkin	97424c05c0	docs: ordering fix	2021-04-24 02:29:30 +03:00
Aliaksandr Valialkin	d7401e40b8	vendor: `make vendor-update`	2021-04-24 01:36:30 +03:00
Aliaksandr Valialkin	2bd99046a1	docs: update docs order	2021-04-24 01:28:13 +03:00
Aliaksandr Valialkin	fecc300e3c	docs/Single-server-VictoriaMetrics.md: mention that the native export format can change between releases Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1203	2021-04-24 01:28:12 +03:00
Aliaksandr Valialkin	cbf545f3af	docs: sync with master branch	2021-04-23 22:05:36 +03:00
Aliaksandr Valialkin	e309b5a83b	app/vmagent/remotewrite: increase the maximum possible number of inmemory blocks for systems with high amounts of RAM This should reduce the probability of using much slower file-based persistent queue when vmagent processes metrics at high rate (millions of metrics per second). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235	2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin	f92db26a93	app/vmagent/remotewrite: count maxLabelsPerBlock as 10x of maxRowsPerBlock This should increase block sizes and subsequently increase the maximum possible bandwidth per each connection to remote storage. This, in turn, should reduce the probability of storing the data in local buffers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235	2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin	320983f650	lib/promscrape: apply `scrape_timeout` on receiving the first response byte for `stream_parse: true` scrape targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047	2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin	34321e5f8d	lib/promscrape/discovery/kubernetes: refresh `role: endpoints` targets on service object removal as Prometheus does This is a follow-up for `ae37cfd528` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:27:29 +03:00
Aliaksandr Valialkin	db27dbab5e	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices targets on service object update like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-04-23 20:12:22 +03:00
Aliaksandr Valialkin	8a6bf55a9b	docs: add missing images for vmbackupmanager.md	2021-04-22 14:46:40 +03:00
Aliaksandr Valialkin	7f573a9564	docs/vmbackupmanager.md: sync with app/vmbackupmanager/README.md	2021-04-22 14:45:31 +03:00
Aliaksandr Valialkin	ab8008d6d7	lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142	2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin	aaee80d158	app/vmbackup: typo fix: snaphsot -> snapshot Follow-up for `9de0fa3649`	2021-04-22 11:18:13 +03:00
Denys Holius	c1fa465fe0	fixed typo; added example of usage vmbackup in docker-compose (#1237 )	2021-04-22 11:18:12 +03:00
Aliaksandr Valialkin	e7c4fde756	app/vmauth: parse `url_prefix` only once during config load	2021-04-21 10:57:17 +03:00
Aliaksandr Valialkin	6dc5d3b357	all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com	2021-04-20 20:20:01 +03:00
Denys Holius	05254a3329	removed DS_Store files from VM_logo.zip (#1233 )	2021-04-20 11:46:57 +03:00
Aliaksandr Valialkin	64f1ddefe5	all: consistency renaming `Victoria Metrics` -> `VictoriaMetrics` VMInsert -> vminsert VMSelect -> vmselect VMStorage -> vmstorage	2021-04-20 11:45:02 +03:00
Aliaksandr Valialkin	8d869d112b	app/vmauth: follow-up for `6a81a89b3d`	2021-04-20 10:59:22 +03:00
Nikolay	7d249d787d	adds query params support for vmauth urlPrefix (#1226 ) * adds query params support for vmauth urlPrefix * Update app/vmauth/example_config.yml * Update app/vmauth/example_config.yml Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-20 10:59:20 +03:00
Aliaksandr Valialkin	b0bd2c0d61	docs/Cluster-VictoriaMetrics.md: sync with cluster README.md	2021-04-19 13:31:23 +03:00
Roman Khavronenko	2b1f6b2373	dashboard: plot avg GC duration instead of quantile `1` for better perception (#1229 )	2021-04-19 13:29:31 +03:00
nemobis	55b205ab37	Typo fixes in README (#1222 )	2021-04-18 14:17:16 +01:00
Aliaksandr Valialkin	bdcd978d56	app/vmctl: update README.md according to `bfecd0fd55`	2021-04-16 12:10:00 +03:00
Denys Holius	2fcb87bbed	removed MACOSX dirrectory from VM_logo.zip (#1224 )	2021-04-16 10:58:51 +03:00
Denys Holius	9896b51ee5	Added some explanation to vmctl docs (#1217 ) Added explanation that vmctl do not make snapshot from Prometheus when run migration data	2021-04-16 10:58:51 +03:00
dereksfoster99	1b6d2467ff	Update PerTenantStatistic.md	2021-04-16 10:58:51 +03:00
Aliaksandr Valialkin	113a612c86	vendor: update github.com/klauspost/compress from v1.11.13 to v1.12.1	2021-04-14 14:22:14 +03:00
Aliaksandr Valialkin	6ea174e78a	docs/Cluster-VictoriaMetrics.md: mention that sometimes `-replicationFactor` shouldnt be set at `vmselect` nodes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207	2021-04-14 13:06:49 +03:00
Aliaksandr Valialkin	4a54c212d6	CONTRIBUTING.md: sync with master branch	2021-04-14 12:52:03 +03:00
Aliaksandr Valialkin	cd1ab50fcc	CODE_OF_CONDUCT.md: sync with master branch	2021-04-14 12:50:26 +03:00
Aliaksandr Valialkin	1088113110	lib/uint64set: remove memory allocation in getOrCreateSmallPool() This partially reverts `fb82c4b9fa` It has been appeared that the additional memory allocation may result in higher GC pauses. It is better to spend CPU time on copying bigger bucket16 structs instead of increasing query latencies due to higher GC pauses	2021-04-14 12:31:53 +03:00
f41gh7	b493937815	updates per-tenant stats docs changes docs order changes per-tenant-stats pic	2021-04-14 12:31:51 +03:00
Artem Navoiev	39b5de9f24	[draft] per tenant statistic (#121 ) * [draft] per tenant statistic * updates metric name update graph adds link and example config * quick fix * adds grafana dashboard adds example alert Co-authored-by: f41gh7 <nik@victoriametrics.com>	2021-04-14 11:23:41 +03:00
Aliaksandr Valialkin	72c41323fa	lib/storage: code clarification: remove caching the found metricName in searchMetricName	2021-04-13 10:20:35 +03:00
Roman Khavronenko	c6fc3fa94d	alerts: make alerting rule `RPCErrors` compatible with PromQL (#1204 ) Original query can't be executed via PromQL which results in error if expression is evaluated by Prometheus. The new expression is compatible with both engines.	2021-04-13 08:10:23 +03:00
Aliaksandr Valialkin	c872ba45b9	docs: update `-help` output after the commit `77be3e3a82`	2021-04-12 12:35:39 +03:00
Artem Navoiev	c3dcfdef8c	improve docs for cli flags (#1202 ) * improve docs for cli flags * improve docs for cli flags.2	2021-04-12 12:28:36 +03:00
Aliaksandr Valialkin	2190fd2148	docs: `make docs-sync`	2021-04-10 19:55:05 +03:00
Artem Navoiev	ddf3f17887	update vmgateway.md (#1201 )	2021-04-10 19:55:04 +03:00
Aliaksandr Valialkin	0f7ece84f3	app/vmstorage/transport: reduce memory allocations on data ingestion path	2021-04-10 17:36:00 +03:00
Lapo Luchini	19906dc30e	Allow specifying build date from a variable (#1200 ) Just like the existing infrastructure for `BUILDINFO_TAG`, this can ease the production of [reproducible builds](https://wiki.freebsd.org/ReproducibleBuilds). (e.g. in FreeBSD the date the port was committed is used at build time, not the actual build time, so that an identical port produced at different times produces an identical executable)	2021-04-10 15:47:53 +03:00
Roman Khavronenko	46e6fdb131	Docs update (#1199 ) * docs: drop table of contents for `vmctl` We already have it autogenerated on .github.io, so no need to keep it. * docs: mention OpenTSDB migration feature for vmctl * docs: sync docs for `vmalert`	2021-04-10 15:47:52 +03:00
John Seekins	9e7bf595a0	Improve documentation on OpenTSDB migration tool and fix a bug with hard offsets (#1198 ) * add more documentation on OpenTSDB migration explaining what chunking means * more clarification of OpenTSDB aggregations * break out what a retention string becomes * add more docs around retention strings * add example of running program and fix mistake in how hard offsets are handled * fix formatting	2021-04-10 15:28:56 +03:00
John Seekins	97fafce028	OpenTSDB migration to VictoriaMetrics (#1089 )	2021-04-10 15:28:54 +03:00
Roman Khavronenko	c4f6b79d76	alerts: add `ServiceDown` alert to detect "dead" services (#1196 )	2021-04-08 18:23:10 +03:00
Roman Khavronenko	712725b4a5	vmalert: document template functions and mention them in README (#1197 )	2021-04-08 18:20:57 +03:00
Aliaksandr Valialkin	c3893805f2	docs/Single-server-VictoriaMetrics.md: recommend using the official alerts for VictoriaMetrics in `Monitoring` section	2021-04-08 12:14:28 +03:00
Aliaksandr Valialkin	3b1f0cb3f6	lib/promscrape: create a single `swosFunc` per `scrape_config`	2021-04-08 09:31:38 +03:00
Aliaksandr Valialkin	084ee9f598	vendor: `make vendor-update`	2021-04-08 01:00:43 +03:00
Aliaksandr Valialkin	3e4f063afc	docs/CHANGELOG.md: cut v1.58.0	2021-04-08 00:48:46 +03:00
Aliaksandr Valialkin	e48fd5776b	app/vmselect/promql: fix tests after `d3fa0ccabd`	2021-04-08 00:17:40 +03:00
Aliaksandr Valialkin	f90bf265f4	app/vmselect/promql: properly detect aggregate `topk` and `bottomk` aggregate functions in order to disable duplicate sorting Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-08 00:10:33 +03:00
Aliaksandr Valialkin	9ce3b7e1dd	app/vmselect: return `data:null` instead of `data:[]` from `/api/v1/query_exemplars`, since Grafana throws an error otherwise Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-07 23:34:01 +03:00
Aliaksandr Valialkin	276dbc2133	lib/promscrape: do not spend CPU time on constructing scrapeWork key if clustering is disabled	2021-04-07 21:55:06 +03:00
Aliaksandr Valialkin	3d5f1f779f	app/vmselect: do not sort series returned from `topk` and `bottomk` functions, since these series are already sorted in user-expected order Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189	2021-04-07 14:16:02 +03:00
Aliaksandr Valialkin	18cad589d7	docs/Cluster-VictoriaMetrics.md: follow-up after `c6a8ebb11f`	2021-04-07 13:46:56 +03:00
Roman Khavronenko	ff3711eea2	docs: update docs ordering and formatting (#1192 ) The major change is adding `sort` directive to docs. For those docs which are copied from internal packages `sort` is added via makefile command. For the rest it is added manually since they're updated manually as well. The rest of changes is connected with markdown formatting. For example, changing headers in some files (`##` => `#`) makes navigation on .github.io to look better. This especially useful for `changelog` docs. Table of contents for `vmctl` is dropped, since we already have it autogenerated on .github.io. No link changes expected. The corresponding PR to `cluster` branch will be made in follow-up PR.	2021-04-07 13:43:01 +03:00
Aliaksandr Valialkin	59ccc43e3a	lib/storage: properly handle big time ranges passed to `/api/v1/labels` and `/api/v1/label/<labelName>/values` It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch	2021-04-07 13:33:10 +03:00
Aliaksandr Valialkin	02b83e0957	lib/promscrape/discovery: remove superflouos check in registerPendingAPIWatchers The check `_, ok := uw.aws[aw]; !ok` isn't needed, since aw cannot exist in uw.aws because of the check inside subscribeAPIWatcher	2021-04-07 13:10:04 +03:00
Aliaksandr Valialkin	db56ee0e28	lib/promscrape/discovery/kubernetes: register pending apiWatchers in uw.aws	2021-04-06 11:11:53 +03:00
Aliaksandr Valialkin	7fce4e9fb4	app/vmselect: return dumb response on `/api/v1/query_exemplars` request Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186	2021-04-05 23:28:21 +03:00
Aliaksandr Valialkin	2585058a5f	Makefile: prepare `arm64` and `amd64` release archives for cluster version on `make release` command	2021-04-05 23:01:45 +03:00
Aliaksandr Valialkin	edd66b7e82	lib/promscrape/discovery/kubernetes: remove superflouos mustStart and mustStop functions	2021-04-05 22:43:49 +03:00
Aliaksandr Valialkin	16e8e09d61	deployment/docker/docker-compose.yml: update Grafana from v7.5.1 to v7.5.2	2021-04-05 22:30:51 +03:00
Roman Khavronenko	51faea5e4b	deployment: add vmalert+alertmanager services and list of default alerts for cluster version (#1187 )	2021-04-05 22:29:04 +03:00
Roman Khavronenko	7fc9239536	Cluster dashboard update (#1185 ) * dashboard: change FreeDiskSpace panel to show percentage of used space instead * dashboard: disable area fill for Cache hit ratio * dashboard: minor display updates * dashboard: add panel `Concurrent flushes on disk` * dashboard: add `Rows ignored` panel * dashboard: update ChurnRate panel with proper description and additional query over 24h time window	2021-04-05 22:28:02 +03:00
Lu Jiajing	4ee6def68b	fix access to nil url.URL (#1180 ) fix access to nil url.URL Signed-off-by: Megrez Lu <lujiajing1126@gmail.com> Update lib/promscrape/discovery/kubernetes/api_watcher.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-05 22:26:43 +03:00
Aliaksandr Valialkin	7eca60694e	lib/promscrape/discovery/kubernetes: reduce CPU time spent on registering big number of Kubernetes objects shared among big number of scrape jobs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 22:05:02 +03:00
Aliaksandr Valialkin	9da2ef3d8f	lib/promscrape/discovery/kubernetes: load objects missing in local cache from api seriver in getObjectByRole() This should fix possible race for `role: endpoints` and `role: endpointslices` service discovery, when the referred `pod` and `service` objects aren't propagated to urlWatcher cache yet. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182#issuecomment-813353359 for details.	2021-04-05 20:31:22 +03:00
Aliaksandr Valialkin	4d6a3aba4d	lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030	2021-04-05 19:26:51 +03:00
Aliaksandr Valialkin	fe084fdd33	lib/promscrape/discovery/kubernetes: synchronously load Kubernetes objects on first access Remove async registration of apiWatchers, since it breaks discovering `role: endpoints` and `role: endpointslices` targets, which depend on pod and service objects. There is no need in reloading `endpoints` and `endpointslices` targets if the referenced `pod` or `service` objects change, since in this case the corresponding `endpoints` and `endpointslices` objects should also change because they contain ResourceVersion of the referenced `pod` or `service` objects, which is modified on object update. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182	2021-04-05 14:37:07 +03:00
Aliaksandr Valialkin	c8b7c32e23	lib/proxy: typo fix after `a5c5b54c22`	2021-04-05 14:37:06 +03:00
Aliaksandr Valialkin	e0b687171b	lib/proxy: add support for `socks5 over tls` proxy	2021-04-05 13:00:42 +03:00
Aliaksandr Valialkin	90da332ea0	lib/promscrape: pass `X-Prometheus-Scrape-Timeout-Seconds` header to scrape targets as Prometheus does	2021-04-05 12:15:14 +03:00
Aliaksandr Valialkin	c229e10619	docs/CHANGELOG.md: explain why `-sortLabels` is set to false by default	2021-04-04 01:59:19 +03:00
Aliaksandr Valialkin	92d4c80639	docs/vmagent.md: mention that vmagent supports scraping via socks5 proxy	2021-04-04 01:45:46 +03:00
Aliaksandr Valialkin	afb139c244	docs/CHANGELOG.md: document the ability to use socks5 proxy Follow-up for `a4c6a3b3e1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1177	2021-04-04 01:44:01 +03:00
Nikolay	d0b664454b	adds socks5 support for fasthttp client (#1178 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1177 Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-04-04 01:43:59 +03:00
Aliaksandr Valialkin	dd19fab7c9	lib/promscrape: properly send full url in `GET` request via simple HTTP proxy This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 01:20:43 +03:00
Aliaksandr Valialkin	ab9e1eb41f	lib/promscrape: support for simple HTTP proxies without `CONNECT` method support such as https://github.com/prometheus-community/PushProx See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-04 00:40:58 +03:00
Aliaksandr Valialkin	27d8a5d2c0	lib/promscrape: add tests for `authorization` config, which has been added in `df148f48b7`	2021-04-03 22:14:03 +03:00
Aliaksandr Valialkin	4aa59cae7c	app/vmselect/promql: do not delete `dst_label` if `src_label` is empty in `label_copy(q, src_label, dst_label)` and `label_move(q, src_label, dst_label)`	2021-04-03 22:06:50 +03:00
Aliaksandr Valialkin	8b2c4fe0ff	docs/CHANGELOG.md: document the change from `3055ab0115` (add ability to pass "label=value" to the third argument to `topk_` and `bottomk_` functions	2021-04-03 21:42:23 +03:00
Aliaksandr Valialkin	1d88c8523d	docs/Single-server-VictoriaMetrics.md: add missing link in heading to `Graphite Render API usage` chapter	2021-04-03 21:34:57 +03:00
Aliaksandr Valialkin	ae5c20a34c	lib/proxy: log response body on non-200 response code This should improve debuggability for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179	2021-04-03 03:03:41 +03:00
Aliaksandr Valialkin	dfec690548	docs/vmgateway.md: update docs	2021-04-03 00:29:57 +03:00
Aliaksandr Valialkin	e1347a91b0	docs/CHANGELOG.md: yet another typo fix	2021-04-03 00:26:02 +03:00
Aliaksandr Valialkin	6b2e496baf	docs/CHANGELOG.md: typo fix	2021-04-03 00:24:05 +03:00
Aliaksandr Valialkin	b9469de410	app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from `topk_` and `bottomk_` functions in the form: `topk_min(N, m, "label=value")`	2021-04-02 23:56:30 +03:00
Aliaksandr Valialkin	262da1c2e5	docs/{vmauth,vmgateway}.md: small fixes	2021-04-02 23:15:40 +03:00
Aliaksandr Valialkin	5edec0e57e	app/vmgateway: publish docs	2021-04-02 23:10:10 +03:00
Aliaksandr Valialkin	4028d692f5	app: do not process non-GET requests on at `/` handler	2021-04-02 22:56:38 +03:00
Aliaksandr Valialkin	db7ae82923	vendor: `make vendor-update`	2021-04-02 22:24:19 +03:00
Aliaksandr Valialkin	89771d082d	app/vmauth: add support for authorization via `Authorization: Bearer <token>`	2021-04-02 22:15:07 +03:00
Aliaksandr Valialkin	6c6174271a	docs/vmagent.md: mention about proxy_authorization section	2021-04-02 21:24:52 +03:00
Aliaksandr Valialkin	87700f1259	lib/promscrape: add support for `authorization` config in `-promscrape.config` as Prometheus 2.26 does See https://github.com/prometheus/prometheus/pull/8512	2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin	2825a1e86d	lib/promscrape: add `follow_redirect` option to `scrape_configs` section like Prometheus does See https://github.com/prometheus/prometheus/pull/8546	2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin	449ee2ace9	deployment/docker: upgrade Go builder from v1.16.2 to v1.16.3 See https://github.com/golang/go/issues?q=milestone%3AGo1.16.3+label%3ACherryPickApproved	2021-04-02 19:23:02 +03:00
Aliaksandr Valialkin	31d3d02d25	app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.	2021-04-02 16:50:02 +03:00
Aliaksandr Valialkin	245eba8896	lib/promscrape/discovery/kubernetes: properly track objects with the same names in multiple namespaces This is a follow-up for `12e4785fe8` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:46:34 +03:00
Aliaksandr Valialkin	eee860f83d	lib/promscrape/discovery/kubernetes: properly discover targets in multiple namespaces Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170	2021-04-02 14:29:24 +03:00
Aliaksandr Valialkin	4e685fb0df	docs/CHANGELOG.md: mention about AWS IAM roles for tasks support for EC2 service discovery Follow-up for `fdb8995642`	2021-04-02 13:11:05 +03:00
Nikolay	00157f3ec6	Adds aws ECS credentials support (#1175 )	2021-04-02 13:11:03 +03:00
Aliaksandr Valialkin	512addc608	app/{vminsert,vmagent}: add `-sortLabels` command-line option for sorting time series labels before ingesting them in the storage This option can be useful when samples for the same time series are ingested with distinct order of labels. For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.	2021-03-31 23:27:21 +03:00
Aliaksandr Valialkin	ae1c653d55	lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels	2021-03-31 21:22:40 +03:00
Aliaksandr Valialkin	392ba94d1d	app/vmagent/remotewrite: reduce memory usage when `-remoteWrite.queues` is set to a big value See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1167	2021-03-31 16:17:23 +03:00
Aliaksandr Valialkin	6712173ce3	docs/Articles.md: add a link to https://blog.kintone.io/entry/2021/03/31/175256	2021-03-31 15:34:57 +03:00
Aliaksandr Valialkin	b873b965af	app/vmagent/remotewrite: reduce memory usage when samples with big number of labels are sent to remote storage	2021-03-31 00:45:42 +03:00
Aliaksandr Valialkin	108bf68a91	docs/Single-server-VictoriaMetrics.md: update `victoria-metrics -help` output after `e7fdea5953`	2021-03-30 21:47:12 +03:00
Aliaksandr Valialkin	d8c769e6af	app/vmselect: add `-search.maxStatusRequestDuration` command-line flag for limiting the duration of requests to `/api/v1/status/*` and `/api/v1/series/count`	2021-03-30 21:39:32 +03:00
Denys Holius	7e6f168fc3	deployment: Grafana version updated to 7.5.1 (#1161 )	2021-03-30 20:44:40 +03:00
Aliaksandr Valialkin	c237a5f0d1	docs/CHANGELOG.md: cut v1.57.1	2021-03-30 15:40:23 +03:00
Aliaksandr Valialkin	4fa8ae1621	docs/CHANGELOG.md: mention about returned back `type` label for `vm_tenant_inserted_rows_total` metric See `9b4e608199`	2021-03-30 15:16:33 +03:00
Aliaksandr Valialkin	0e29d48628	app/vmselect: remove mentioning for `-search.storageTimeout` from command-line flag descriptions This is a follow-up for `ecfd6fe78d`	2021-03-30 15:07:40 +03:00
Aliaksandr Valialkin	ecfd6fe78d	app/vmselect: remove `-search.storageTimeout` command-line flag, since it has the same meaning as `-search.maxQueryDuration` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 14:54:36 +03:00
Aliaksandr Valialkin	7bafaad46d	app/vmselect: prevent from possible incomplete query results after timed out query Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 13:34:57 +03:00
Aliaksandr Valialkin	9b4e608199	app/vminsert: return back `type` label to per-tenant metric `vm_tenant_inserted_rows_total The `type` label has been removed in the commit `022fe4efd0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932	2021-03-30 00:06:59 +03:00
Aliaksandr Valialkin	a9e0c09be0	Makefile: build vmutils for arm on `make release-vmutils` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1147	2021-03-29 23:16:30 +03:00
Aliaksandr Valialkin	0a8f0a4e2f	all: increase minimum supported Go version for building VictoriaMetrics components from v1.14 to v1.15 This is needed after the commit `c0ac740f93`, which uses URL.Redacted() method, which has been added in v1.15. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1147	2021-03-29 23:06:36 +03:00
Aliaksandr Valialkin	6bfac94995	docs/CHANGELOG.md: cut v1.57.0	2021-03-29 19:15:01 +03:00
Aliaksandr Valialkin	8c957f0483	docs/CHANGELOG.md: typo fixes	2021-03-29 15:47:03 +03:00
Aliaksandr Valialkin	842c5a6202	docs/CHANGELOG.md: mention about logging of metrics with too old timestamps in a single-node VictoriaMetrics This is a follow up for `aa81039b42`	2021-03-29 15:43:27 +03:00
Aliaksandr Valialkin	3b8894e51e	docs/CHANGELOG.md: mention Graphite Render API fixes	2021-03-29 14:28:25 +03:00
Aliaksandr Valialkin	3fdc43a1c9	docs/CHANGELOG.md: mention optimized query performance on systems with many CPU cores	2021-03-29 13:55:46 +03:00
Roman Khavronenko	b736c40053	Dashboards update (#1153 ) * dashboard: update single node dashboard * add number of new series created over last 24h; * bump version requirements. * dashboard: update vmagent dashboard * add panel for open file descriptors; * add panel for disk I/O; * add panel for `vmagent_remotewrite_packets_dropped_total` metric; * bump version requirements.	2021-03-29 12:41:28 +03:00
Aliaksandr Valialkin	887e67f13b	lib/uint64set: improve Set.Has() performance scalability on multi-CPU system Do not update bucket32.hint on Set.Has() call, since it leads to memory ping-pong between CPU cores multi-CPU system	2021-03-29 12:34:19 +03:00
Aliaksandr Valialkin	3f0534c7f8	app/vminsert/opentsdbhttp: remove rowsInsertedTenant metric, since it is substituted by tenantmetrics.RowsInsertedByTenant in the commit `022fe4efd0`	2021-03-29 12:16:13 +03:00
Aliaksandr Valialkin	940a547116	lib/storage: do not update b.nextIdx if no samples are removed because of retention	2021-03-29 12:13:38 +03:00
Nikolay	022fe4efd0	makes tenant metrics names consistent with enterprise version (#1152 )	2021-03-29 11:59:04 +03:00
Aliaksandr Valialkin	8a3939e93e	docs: document that vmagent drops data blocks when remote storage replies with 400 and 409 http status codes This is a follow up for `1b7dc1e5a5`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149	2021-03-26 14:43:41 +02:00
Aliaksandr Valialkin	3fedf680f8	app/vmagent/remotewrite: remove superflouos code after `1b7dc1e5a5`	2021-03-26 14:01:37 +02:00
Aliaksandr Valialkin	e71f9fc703	vendor: update github.com/klauspost/compress from v1.11.12 to v1.11.13	2021-03-26 14:01:37 +02:00
Nikolay	15ff1f3a94	Adds blocks drop (#1151 ) * adds blocks drop at 400 BadRequest status code recieved from remote storage, not expected that remote storage will be able to handle it on retry * removes error logging for dropped blocks, its expected error	2021-03-26 14:01:37 +02:00
Aliaksandr Valialkin	7d87d42a91	lib/promscrape/discovery/kubernetes: typo fix in error message	2021-03-26 12:46:33 +02:00
Aliaksandr Valialkin	a920e71809	lib/promscrape/discovery/kubernetes: properly handle `too old resource version` error message from Kubernetes watch API	2021-03-26 12:28:35 +02:00
Aliaksandr Valialkin	9c2be144cf	app/vmselect: log the metric which trigger rollup result cache reset This should help finding the source of stale metrics	2021-03-25 21:32:28 +02:00
Aliaksandr Valialkin	c5618949a0	docs/Cluster-VictoriaMetrics.md: mention that vmselect doesnt serve partial responses from export API Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148	2021-03-25 21:03:32 +02:00
Aliaksandr Valialkin	698ccca6ad	app/vmselect/prometheus: explain why partial responses are disallowed during data export See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148	2021-03-25 18:39:07 +02:00
Aliaksandr Valialkin	4ee2f83bda	vendor: `make vendor-update`	2021-03-25 17:59:31 +02:00
Aliaksandr Valialkin	b91e18af0e	docs/vmagent.md: add an example for `-remoteWrite.label`	2021-03-25 17:59:30 +02:00
Aliaksandr Valialkin	82047be90b	docs: add a link to the repository from build instruction for all the VictoriaMetrics components	2021-03-25 17:16:55 +02:00
Aliaksandr Valialkin	6de36b4e21	docs/vmagent.md: cosmetic fixes	2021-03-25 17:11:10 +02:00
Aliaksandr Valialkin	1b13d02728	docs/vmagent.md: cosmetic fixes	2021-03-25 16:54:37 +02:00
Aliaksandr Valialkin	155cd90fc8	docs/vmagent.md: typo fix: tupically -> typically	2021-03-25 16:48:27 +02:00
Aliaksandr Valialkin	450e23533d	docs/vmalert.md: remove misleading `-evaluationInterval=3s` from example config args 3s evaluation interval is too small for practical setups. It can result in increased load on datasource. So it is better to remove it from example config args, which are usually copy-pasted by novice users.	2021-03-25 15:31:10 +02:00
Aliaksandr Valialkin	b473c21915	app/vmselect/promql: do not merge time series during requests to `/api/v1/query` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141	2021-03-25 13:56:23 +02:00
Aliaksandr Valialkin	f971fe86cd	lib/storage: tune loopsCountPerMetricNameMatch according to production workload	2021-03-25 13:48:17 +02:00
Aliaksandr Valialkin	6b1f807418	app/vmagent: add `-promscrape.consul.waitTime` command-line flag for configuring Consul service discovery wait time See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144	2021-03-23 19:34:12 +02:00
Aliaksandr Valialkin	3bcad1c513	docs/CHANGELOG.md: mention the feature from `44a6cc5eca`	2021-03-23 19:00:51 +02:00
Aliaksandr Valialkin	9947c65df3	lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock This should speed up Search.NextMetricBlock a bit	2021-03-23 17:59:34 +02:00
Aliaksandr Valialkin	2041008d64	app/vmagent: mention -remoteWrite.maxDiskUsagePerURL in the descriptio of `-remoteWrite.tmpDataPath` flag	2021-03-23 16:38:41 +02:00
Nikolay	19a40faf8e	changes consul_service label value (#1143 ) according to prometheus discovery. It should mitigate issue with case sensetive services https://github.com/hashicorp/consul/issues/5707	2021-03-23 15:37:06 +02:00
Aliaksandr Valialkin	d1e773266f	app/vmselect/graphite: accept and enforce `extra_label` in all the Graphite APIs	2021-03-23 15:30:15 +02:00
Aliaksandr Valialkin	27bb614016	app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon	2021-03-23 14:57:57 +02:00
Aliaksandr Valialkin	12ca0efc19	lib/storage: respect the deadline passed to Storage.SearchMetricNames	2021-03-22 23:03:00 +02:00
Aliaksandr Valialkin	40e47935e7	lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache	2021-03-22 23:02:59 +02:00
Aliaksandr Valialkin	e01bf33485	docs/Single-server-VictoriaMetrics.md: sync with README.md	2021-03-22 17:52:14 +02:00
Aliaksandr Valialkin	020b031bc3	docs/Articles.md: add https://blog.cybozu.io/entry/2021/03/18/115743	2021-03-22 17:52:13 +02:00
Aliaksandr Valialkin	7f9863254d	app/vmselect: improve description for `-search.maxPointsPerTimeseries` command-line flag	2021-03-22 16:44:36 +02:00
Aliaksandr Valialkin	6ebf18ab97	app/{vminsert,vmagent}: use Influx field as metric name if measurement is empty and -influxSkipSingleField command-line is set See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139	2021-03-22 13:55:11 +02:00
Aliaksandr Valialkin	1618b0ca6d	lib/storage: tune loopsCountPerMetricNameMatch	2021-03-22 12:57:54 +02:00
Aliaksandr Valialkin	92531a38c4	app/vmselect/promql: increment key prefix for faster reset for rollup result cache	2021-03-22 11:59:39 +02:00
Aliaksandr Valialkin	0b358ee150	vendor: update github.com/VictoriaMetrics/metrics from v1.17.0 to v1.17.1	2021-03-18 18:53:40 +02:00
Aliaksandr Valialkin	799461d8bf	app/vmselect/graphite: follow-up after `529d7be26b`	2021-03-18 16:33:00 +02:00
Nikolay	8807410a00	changes metricsFind api (#1137 ) it should be able mitigate crash if label value contains *,[ or { symbols	2021-03-18 16:32:59 +02:00
Aliaksandr Valialkin	ff6fe0698b	docs/Single-server-VictoriaMetrics.md: remove outdated message about experimental mode for `-pure` builds	2021-03-18 16:16:33 +02:00
Aliaksandr Valialkin	7503111feb	lib/storage: small code simplification after `6cee5338b2`	2021-03-18 15:22:39 +02:00
Aliaksandr Valialkin	4443254fb9	lib/storage: prevent from infinite loop if `{__graphite__="..."}` filter matches a metric name with `*`, `[` or `{` chars The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137	2021-03-18 14:57:39 +02:00
Aliaksandr Valialkin	e03233f441	lib/fs: reduce the frequency of `failed to remove directory ... due to NFS lock` log warnings Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.	2021-03-18 13:23:43 +02:00
Roman Khavronenko	540c00f2a0	dashboard: update cluster node dashboard (#1136 ) * add panel `Open FDs` for file descriptors metrics; * add panel `Disk writes/reads` to show the real read/write load on storage layer; * add stats panel to show available CPUs, memory and disk space.	2021-03-18 12:04:12 +02:00
Aliaksandr Valialkin	156b4c2490	vendor: update github.com/VictoriaMetrics/metrics from v1.16.0 to v1.17.0	2021-03-17 23:22:40 +02:00
Aliaksandr Valialkin	93a318d1bf	vendor: update github.com/VictoriaMetrics/metrics from v1.15.3 to v1.16.0 This adds the following new metrics for each VictoriaMetrics app: * process_resident_memory_anonymous_bytes - the RSS share for memory allocated by the process itself. This share cannot be freed by the OS, so it must be taken into account by OOM killer. * process_resident_memory_pagecache_bytes - the RSS share for page cache memory (aka memory-mapped files). This share can be freed by the OS at any time, so it must be ignored by OOM killer.	2021-03-17 17:59:31 +02:00
Aliaksandr Valialkin	b859fe7879	lib/storage: faster move heavy filters to the end of list	2021-03-17 15:11:56 +02:00
Aliaksandr Valialkin	667d129e1e	app/vmselect/promql: typo fix after `9666834045`	2021-03-17 15:11:54 +02:00
Aliaksandr Valialkin	69201806f8	app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in `buckets_limit()` function This should improve accuracy for the returned buckets	2021-03-17 14:31:29 +02:00
Aliaksandr Valialkin	a3b3d434a3	docs/CHANGELOG.md: cut v1.56.0	2021-03-17 02:05:10 +02:00
Aliaksandr Valialkin	265ac5f695	docs/CHANGELOG.md: do not mention reduction in query duration There was a signficant refactoring in the code responsible for time series search, so it can result in both speed ups and slow downs depending on used queries.	2021-03-17 01:58:34 +02:00
Aliaksandr Valialkin	8ef1184adf	app/vmstorage: add `vm_index_search_duration_seconds` histogram for monitoring the performance of index search	2021-03-17 01:13:15 +02:00
Aliaksandr Valialkin	5e77a939c2	all: make `go vet` happy	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	41fe707bec	lib/storage: limit loops count in order to reduce max CPU usage during filter search	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	e2a0c8bd72	lib/storage: do not modify filterLoopsCount stats with loopsCount stats Such a modification can result in incorrect filter sorting later	2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin	b997f4a418	all: make golangci-lint happy after the commit `6378205415`	2021-03-17 00:24:31 +02:00
Aliaksandr Valialkin	8005ba26b9	lib/netutil: enable IPv6 UDP listening if `-enableTCP6` command-line flag is passed to VictoriaMetrics This is a follow-up for `18cfc4be7b` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:19:30 +02:00
Nikolay	b293f81eb8	Adds udp6 support for ingest servers (#1134 ) with flag -enableUDP6 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:19:29 +02:00
Aliaksandr Valialkin	05e301cfa0	app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores	2021-03-16 21:52:05 +02:00
Aliaksandr Valialkin	1cfe3f872f	lib/uint64set: optimize bucket16.addMulti a bit	2021-03-16 21:08:38 +02:00
Aliaksandr Valialkin	3caac5edd4	Makefile: prepare vmutils-windows-.zip archive on `make release-vmutils` command The archive contains the following executables for Windows: vmagent * vmalert * vmauth * vmctl Other components - vmbackup, vmrestore, victoria-metrics - aren't supported for Windows yet	2021-03-16 20:54:10 +02:00
Aliaksandr Valialkin	727ded9d4e	lib/storage: time series search optimization according to production workload profiling Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down the function by multiple times when it finds big number of metricIDs (tens of millions).	2021-03-16 20:08:43 +02:00
Aliaksandr Valialkin	cb1d1648a2	vendor: `make vendor-update`	2021-03-16 19:07:22 +02:00
Aliaksandr Valialkin	f4a44d6c0d	lib/storage: further tuning for time series search	2021-03-16 18:47:29 +02:00
Aliaksandr Valialkin	f54ece438d	app/vmselect/promql: do not crash if `histogram_over_time()` function name contains uppercase letters such as `Histogram_over_time()`	2021-03-16 12:23:55 +02:00
Aliaksandr Valialkin	8720078916	vendor: update github.com/VictoriaMetrics/metrics from v1.15.2 to v1.15.3	2021-03-16 12:16:20 +02:00
Aliaksandr Valialkin	5c28e76ef8	docs/Articles.md: added a link to https://www.youtube.com/watch?v=QgLMztnj7-8	2021-03-15 23:02:35 +02:00
Aliaksandr Valialkin	d074326970	app/vmstorage: add `-logNewSeries` command-line flag for determining the source of series churn rate	2021-03-15 22:40:28 +02:00
Aliaksandr Valialkin	c4d0c20de2	lib/influxutils: return response compatible with InfluxDB 1.8.4 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:20:38 +02:00
f41gh7	b7c9f3676e	typo fix	2021-03-15 22:20:37 +02:00
Aliaksandr Valialkin	e2717d84c0	all: various fixes in command-line flag descriptions	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	776b8b32ca	app/{vminsert,vmagent}: a follow-up for `b1aa8c3d8f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:03:49 +02:00
Nikolay	a843dc0219	adds fake response for telegraph queries (#1130 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin	fc902734d9	lib/storage: further tuning for time series selector code	2021-03-15 20:32:37 +02:00
Aliaksandr Valialkin	30ab0eec27	app/vmselect/promql: reduce overhead on scrape interval estimation It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.	2021-03-15 20:32:35 +02:00
Aliaksandr Valialkin	8844b38745	app/vmselect/promql: fix tests after `2dae0a2c47`	2021-03-15 20:20:27 +02:00
Aliaksandr Valialkin	dc70e7de76	lib/uint64set: reduce the size of bucket16 by storing smallPool by pointer. This reduces CPU time spent on bucket16 copying inside bucket13.addBucketAtPos.	2021-03-15 17:27:03 +02:00
Aliaksandr Valialkin	cc6b1a5941	lib/uint64set: optimize Set.AddMulti for large sorted sets	2021-03-15 17:10:54 +02:00
Aliaksandr Valialkin	f023e92f1a	lib/uint64set: optimize bucket16.add and bucket16.addMulti a bit	2021-03-15 16:58:24 +02:00
Aliaksandr Valialkin	b7ce96548c	docs/CHANGELOG.md: typo fix: FATURE -> FEATURE	2021-03-15 14:59:40 +02:00
Aliaksandr Valialkin	1c26020080	lib/storage: tune per-day index search	2021-03-15 13:36:36 +02:00
Aliaksandr Valialkin	6b9bba7448	app/vmselect: add `round_digits` query arg to `/api/v1/query` and `/api/v1/query_range` handlers for limiting the number of decimal digits after the point	2021-03-15 12:38:59 +02:00
Aliaksandr Valialkin	1f2a7c3863	docs/Cluster-VictoriaMetrics.md: sync with cluster README.md	2021-03-15 11:51:03 +02:00
Aliaksandr Valialkin	7f52aae20c	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics Do not cache too big byte buffers and too big writeRequestCtx objects, since it is cheaper to re-create them instead of wasting RAM for their caching. This reverts `7f6f350ee1`	2021-03-15 11:49:29 +02:00
sequix	df10d80257	fix vmbackup url in readme	2021-03-15 09:55:11 +02:00
Aliaksandr Valialkin	33cd6c26d3	lib/promscrape: return back the logic for flushing big buffers to storage from the commit `3fd8653b40` This should reduce memory usage when vmagent scrapes targets with big number of metrics and `-promscrape.streamParse` isn't enabled	2021-03-14 22:25:37 +02:00
Aliaksandr Valialkin	894246176f	lib/promscrape/discovery/kubernetes: do not start object watcher until initial objects are loaded	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	9e55db4a53	lib/promscrape: retry service discovery in a few seconds if it starts returning 0 targets This should reduce recovery time from temporary issues during service discovery	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	3b46ae1c05	lib/promscrape: remove duplicate `target` word in error message	2021-03-14 21:56:16 +02:00
Aliaksandr Valialkin	b0b28eeb93	lib/promscrape/discovery/kubernetes: further optimize kubernetes service discovery for the case with many scrape jobs Do not re-calculate labels per each scrape job - reuse them instead for scrape jobs with identical Kubernetes role	2021-03-14 21:16:41 +02:00
Aliaksandr Valialkin	620f05cd2c	lib/promscrape/discovery: fixes after `133b288681` - Removed a deadlock in addAPIWatcher - Do not create unused ScrapeWork objects - Do not spend CPU resources on creating objectByKey map in addAPIWatcher This work is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1125	2021-03-13 15:22:38 +02:00
Aliaksandr Valialkin	afa5b58c2d	app/vminsert/opentsdbhttp: typo fix in `vm_rows_inserted_total{type="opentsdbhttp"}` metric: opentsdb-http -> opentsdbhttp	2021-03-12 15:59:31 +02:00
Aliaksandr Valialkin	b6dcb37fca	app/vmselect/prometheus: do not include datapoints with timestamps matching `t-d` when returning results from `/api/v1/query?query=m[d]&time=t` as Prometheus does	2021-03-12 12:18:37 +02:00
Aliaksandr Valialkin	a12d2013d5	docs/CaseStudies.md: fix incorrect number of active time series for Zhihu	2021-03-12 11:45:52 +02:00
Aliaksandr Valialkin	54f902467d	lib/proxy: there is no need in cloning tlsCfg, which has been created two lines above	2021-03-12 10:48:01 +02:00
Aliaksandr Valialkin	72a8fa484b	lib/proxy: set proxy address in tls.Config.ServerName instead of the target address Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 10:41:25 +02:00
Aliaksandr Valialkin	60e0280a94	lib/promscrape: add ability to configure proxy options via `proxy_tls_config`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 03:36:11 +02:00
Aliaksandr Valialkin	5c7ef14273	deployment/docker: update Go builder from 1.16.1 to 1.16.2 See https://github.com/golang/go/issues?q=milestone%3AGo1.16.2+label%3ACherryPickApproved	2021-03-12 01:53:42 +02:00
Aliaksandr Valialkin	b240d41ede	docs/Articles.md: add https://www.sensedia.com/post/monitoring-with-prometheus-alertmanager	2021-03-12 01:23:22 +02:00
Aliaksandr Valialkin	b2732575f7	lib/storage: further tune filters sorting logic	2021-03-12 00:51:35 +02:00
Aliaksandr Valialkin	8fc29ffc67	lib/promscrape/discovery/kubernetes: use a single watcher per apiURL Previously multiple scrape jobs could create multiple watchers for the same apiURL. Now only a single watcher is used. This should reduce load on Kubernetes API server when many scrape job configs use Kubernetes service discovery.	2021-03-11 17:04:14 +02:00
Aliaksandr Valialkin	19af3b4f38	deployment/docker: update Go builder from 1.16.0 to 1.16.1 See https://github.com/golang/go/issues?q=milestone%3AGo1.16.1+label%3ACherryPickApproved	2021-03-11 17:04:14 +02:00
Aliaksandr Valialkin	8b8d4cbcfe	lib/proxy: do not show inline basic auth passwords when logging errors related to `proxy_url`	2021-03-11 13:44:14 +02:00
Aliaksandr Valialkin	41f641b132	lib/promscrape/discovery/kubernetes: localize Bookmark parsing code This is a follow-up for `e772d1c920`	2021-03-11 13:08:56 +02:00
Aliaksandr Valialkin	ccfe944ce7	docs/ExttendedPromQL.md: remove outdated doc	2021-03-11 12:42:57 +02:00
Brensted	03c8438099	Update BestPractices.md (#1123 ) update lists, hyperlinks fixed.	2021-03-11 12:42:56 +02:00
Aliaksandr Valialkin	6c9cd3f7c1	lib/promscrape/discovery/kubernetes: reduce load on Kubernetes API server by using watch bookmarks This allows continuing object watch from the last bookbark instead of reloading all the objects on watch errors or timeouts. See https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks	2021-03-10 15:08:40 +02:00
Aliaksandr Valialkin	bd8b7a88a7	lib/httpserver: export `vm_available_memory_bytes` and `vm_available_cpu_cores` metrics These metrics are useful for tracking the available memory and CPU cores for VictoriaMetrics apps.	2021-03-10 12:08:26 +02:00
Ihor Borodin	933de6b9b1	Fixing examples of external.alert.source in documentation (#1120 ) * Fixing examples of external.alert.source in documentation	2021-03-10 12:08:22 +02:00
Aliaksandr Valialkin	f05f7831b8	docs/BestPractices.md: initial add	2021-03-10 12:08:17 +02:00
Aliaksandr Valialkin	1042a4897b	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.13 to v1.0.14	2021-03-09 21:34:00 +02:00
Aliaksandr Valialkin	9dfa71ad15	docs/CHANGELOG.md: mention about the bugfix from `787242d7b0`	2021-03-09 20:56:48 +02:00
Aliaksandr Valialkin	e15f3f4f2a	lib/proxy: pass proxy hostname in `Host` header of the `CONNECT` request This should resolve the following issue when connecting to tls proxy: cannot validate certificate for ... because it doesn't contain any IP SANs	2021-03-09 20:41:18 +02:00
Aliaksandr Valialkin	9d8223eafb	lib/proxy: set missing ServerName in TLS config for `proxy_url`. While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.	2021-03-09 19:01:14 +02:00
Nikolay	1310f84122	Changes tlsConfig init for proxy connections (#1121 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-09 19:01:13 +02:00
Aliaksandr Valialkin	0554430d7e	lib/promscrape: apply `sample_limit` after metric relabeling is applied as Prometheus does See the description for `sample_limit` option from Prometheus docs: Per-scrape limit on number of scraped samples that will be accepted. If more than this number of samples are present after metric relabeling the entire scrape will be treated as failed. 0 means no limit. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-03-09 15:52:41 +02:00
Aliaksandr Valialkin	7b66c8cbf8	lib/promscrape/discovery/kubernetes: remove too verbose logs about starting and stopping the watchers Log the number of objects loaded per each watch url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-09 15:07:12 +02:00
Aliaksandr Valialkin	058aa0de75	docs/CHANGELOG.md: mention about improved query performance at `18fe0ff14b`	2021-03-09 13:12:18 +02:00
Aliaksandr Valialkin	942890b1bb	app/vmselect/promql: follow up for `433fff0006`	2021-03-09 12:54:23 +02:00
Nikolay	28e450cd7c	duplicate timeseries fix for prometheus_buckets function (#1119 ) * try fix for prometheus_buckets * merge possible end of the bucket collision	2021-03-09 12:54:21 +02:00
Aliaksandr Valialkin	75d49ee58a	vendor: `make vendor-update`	2021-03-09 12:03:55 +02:00
Aliaksandr Valialkin	2c5ac00231	vendor: update github.com/VictoriaMetrics/fasthttp from 1.0.12 to 1.0.13 This should fix a bug in vmagent with high CPU usage during failed scrapes with small `scrape_timeout`.	2021-03-09 11:44:44 +02:00
John Belmonte	edf39aa225	spelling fix: adjacent (#1115 )	2021-03-09 09:19:16 +02:00
Aliaksandr Valialkin	502fab797a	lib/promscrape: add `scrape_offset` option to `scrape_config` This option can be used for specifying the particular offset per each scrape interval for target scraping	2021-03-08 11:59:32 +02:00
Aliaksandr Valialkin	c4a0bd5eac	lib/storage: go fmt	2021-03-08 11:59:31 +02:00
Aliaksandr Valialkin	c76a904bb0	lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload	2021-03-07 21:17:48 +02:00
Aliaksandr Valialkin	c04505e585	lib/promscrape/discovery/kubernetes: reduce memory usage further when big number of scrape jobs are configured for the same `kubernetes_sd_config` role Serialize reloading per-role objects, so they don't occupy too much memory when objects for many scrape jobs are simultaneously refreshed. Do not reload per-role objects if they were already refreshed by concurrent goroutines. This should reduce load on Kubernetes API server when big number of scrape jobs are configured for the same Kubernetes role. This is a follow-up for `17b87725ed` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-07 20:03:22 +02:00
Aliaksandr Valialkin	175466bb41	lib/decimal: prevent exponent overflow when processing values close to zero See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1114	2021-03-05 18:53:41 +02:00
Aliaksandr Valialkin	26cb6f8861	app/vmauth: allow using regexps in `url_map` paths See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112	2021-03-05 18:53:40 +02:00
Aliaksandr Valialkin	5807ff57f3	lib/promscrape/discovery/kubernetes: reduce memory usage when Kubernetes service discovery is configured on a big number of scrape jobs Previously vmagent was creating a separate Kubernetes object cache per each scrape job. This could result in increased memory usage when monitoring a Kubernetes cluster with big number of objects (pods / nodes / services, etc.) as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113 Now it uses a shared map of scrape objects across multiple scrape jobs.	2021-03-05 17:32:33 +02:00
Aliaksandr Valialkin	92ddb8f197	lib/promscrape/discovery/kubernetes: move apiWatcher code to a separate file	2021-03-05 17:32:32 +02:00
Aliaksandr Valialkin	1c3c844b38	deployment/docker: update base Docker image from alpine:3.13.1 to alpine:3.13.2 See https://www.alpinelinux.org/posts/Alpine-3.13.2-released.html	2021-03-05 10:35:33 +02:00
Aliaksandr Valialkin	dfac6b53f5	docs/Articles.md: add https://dalefro.medium.com/prometheus-victoria-metrics-on-aws-ecs-62448e266090	2021-03-05 09:07:22 +02:00
Aliaksandr Valialkin	02c0959380	lib/promscrape: make cluster membership calculations consistent across 32-bit and 64-bit architectures	2021-03-05 09:06:08 +02:00
Aliaksandr Valialkin	5277507932	app/vmselect/promql: add `histogram_avg()`, `histogram_stddev()` and `histogram_stdvar()` functions to MetricsQL	2021-03-04 14:12:52 +02:00
Aliaksandr Valialkin	133fb9fc00	lib/promscrape: add `-promscrape.cluster.replicationFactor` command-line flag for replicating scrape targets among `vmagent` instances in the cluster	2021-03-04 10:21:27 +02:00
Aliaksandr Valialkin	0ce89b85d4	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics needs free resources for handling workload spikes	2021-03-04 10:21:26 +02:00
Aliaksandr Valialkin	6b596885b4	vendor: sync with `go mod vendor`	2021-03-03 23:03:00 +02:00
Aliaksandr Valialkin	bae7a1b47a	lib/promscrape/discovery/kubernetes: fix tests after `e154f4a644`	2021-03-03 22:42:04 +02:00
Aliaksandr Valialkin	d109e17f46	all: bump minimum supported Go version from 1.13 to 1.14	2021-03-03 15:58:17 +02:00
Aliaksandr Valialkin	84d7ece824	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.7 to v1.5.8	2021-03-03 15:52:42 +02:00
Aliaksandr Valialkin	3c11ce9356	docs/vmagent.md: remove outdated suggestion for determining labels that lead to duplicate targets The original labels for duplicate targets is already printed in the error message starting from `71ea4935de`	2021-03-03 12:28:50 +02:00
Aliaksandr Valialkin	0b8d9350d4	docs/CHANGELOG.md: cut v1.55.1 release	2021-03-03 11:49:26 +02:00
Aliaksandr Valialkin	b178f96ac2	docs/vmagent.md: `make docs-sync` after the commit `621bf03745`	2021-03-03 10:53:43 +02:00
Roman Khavronenko	219291e084	Vmagent docs upd (#1104 ) * vmagent: port changes from https://github.com/VictoriaMetrics/VictoriaMetrics.github.io/pull/1 Thanks to @dereksfoster99 for this patch! * vmagent: reword to make the meaning clear	2021-03-03 10:53:41 +02:00
Aliaksandr Valialkin	3a79fa147b	docs/CHANGELOG.md: mention recent bugfixes from commits `7906316741` and `e154f4a644`	2021-03-03 10:50:59 +02:00
Nikolay	7d92ef3acd	Fix ingress discovery api (#1110 )	2021-03-03 10:45:50 +02:00
Aliaksandr Valialkin	2ecee0515a	app/vmalert/README.md: sync with docs/vmalert.md	2021-03-03 10:42:54 +02:00
Aliaksandr Valialkin	25f453ce1a	lib/promscrape/discovery/kubernetes: properly check for nil pointer inside interface See https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1 This fixes a panic when the ScrapeWork is filtered out in swcFunc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1108	2021-03-03 10:42:54 +02:00
Alexandros Orfanos	ff5b8346d4	vmalert: docs update - status endpoint needs group ID, not group name (#1106 )	2021-03-03 10:42:54 +02:00
Aliaksandr Valialkin	3a1d884618	docs/CHANGELOG.md: cut v1.55.0	2021-03-02 21:39:53 +02:00
Aliaksandr Valialkin	34dec59204	vendor: `make vendor-update`	2021-03-02 21:39:52 +02:00
Aliaksandr Valialkin	fea27b9845	lib/promscrape: go fmt	2021-03-02 21:20:23 +02:00
Aliaksandr Valialkin	c67a07b469	lib/handshake: log read/write operation duration on connection errors This improve debuggability of network errors	2021-03-02 21:20:20 +02:00
Aliaksandr Valialkin	c8dde1fd6b	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:48:44 +02:00
Aliaksandr Valialkin	3fbe2bf1c8	lib/promscrape: pre-allocate space for a map in mergeLabels This should reduce the number of memory allocations when discovering big number of targets	2021-03-02 18:42:44 +02:00
Aliaksandr Valialkin	ac5c47a9f5	lib/promscrape/discovery: properly track vm_promscrape_discovery_kubernetes_objects_removed_total metric	2021-03-02 18:33:29 +02:00
Aliaksandr Valialkin	62d7e07ff7	lib/promrelabel: remove unneded optimizations for `labeldrop` and `labelkeep` actions These optimizations may slow down code execution by matching the same label against regexp two times instead of a single time	2021-03-02 18:01:08 +02:00
Aliaksandr Valialkin	f9c1fe3852	lib/promscrape/discovery/kubernetes: cache ScrapeWork objects as soon as the corresponding k8s objects are changed This should reduce CPU usage and memory usage when Kubernetes contains tens of thousands of objects	2021-03-02 16:44:19 +02:00
Aliaksandr Valialkin	f686174329	lib/promscrape/discovery/ec2: follow-up after `f6114345de`	2021-03-02 13:47:35 +02:00
Nikolay	fd8ca7df50	Adds webIndentity token for aws (#1099 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1080	2021-03-02 13:47:34 +02:00
Aliaksandr Valialkin	e45c399467	lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.	2021-03-02 13:22:10 +02:00
Aliaksandr Valialkin	f4969a624d	lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as `foo; 1 2` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100	2021-03-01 17:17:01 +02:00
Aliaksandr Valialkin	d9e8af0e8f	docs: actualize `-help` output	2021-03-01 17:02:05 +02:00
Aliaksandr Valialkin	3a87d38912	docs/CHANGELOG.md: mention the out of range panic bugfix `d6a41b6ea2`	2021-03-01 16:54:18 +02:00
Aliaksandr Valialkin	5e1a0899b9	vendor: update github.com/VictoriaMetrics/metrics from v1.15.1 to v1.15.2 This should fix an edge case panic when 1e-9 value is passed to Histogram.Update See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1096	2021-03-01 16:49:31 +02:00
Aliaksandr Valialkin	b89a4fac2f	lib/promscrape/discovery/kubernetes: deflake tests; a follow-up for `05fb08713c`	2021-03-01 14:31:44 +02:00
Aliaksandr Valialkin	c3bf72992f	lib/promscrape: explicitly stop and cleanup service discovery routines when new config is read from `-promscrape.config` This should reduce memory usage when `-promscrape.config` file frequently changes	2021-03-01 14:15:16 +02:00
Aliaksandr Valialkin	8af9370bf2	lib/promscrape: use target arg in ScrapeWork cache	2021-03-01 12:31:11 +02:00
Aliaksandr Valialkin	402543e7c6	.github/workflows/main.yml: update Go version from v1.15 to v1.16	2021-03-01 12:14:14 +02:00
Aliaksandr Valialkin	5328506c3e	lib/promscrape: typo fix, which prevented from caching ScrapeWork entries	2021-03-01 12:13:13 +02:00
Aliaksandr Valialkin	d5058caccc	lib/promscrape: add vm_promscrape_scrapework_cache_* metrics for tracking ScrapeWork cache effectiveness	2021-03-01 12:05:58 +02:00
Aliaksandr Valialkin	323af49234	app/vmagent/remotewrite: clarify `-remoteWrite.flushInterval` flag description	2021-03-01 11:51:08 +02:00
Aliaksandr Valialkin	57d2a27a64	docs/CHANGELOG.md: mention the issue related to using Kubernetes watch API for service discovery	2021-03-01 01:42:05 +02:00
Aliaksandr Valialkin	b8dd5e8292	app/vmagent/remotewrite: fix rate limiting logic for `-remoteWrite.url`	2021-03-01 00:59:57 +02:00
Aliaksandr Valialkin	ee5d26a546	lib/httpserver: make `make errcheck` happy after the commit `9fc7726d84`	2021-03-01 00:35:30 +02:00
Aliaksandr Valialkin	d0be1f6f49	app/vmagent: remove data race when applying rate limits to `-remoteWrite.url` with multiple queues	2021-03-01 00:28:20 +02:00
Aliaksandr Valialkin	109bfaadad	lib/promscrape: reduce CPU usage an memory allocations when constructing scrapeWorkKey	2021-02-28 22:30:36 +02:00
Aliaksandr Valialkin	17eb29206d	docs/vmbackup.md: clarify docs on `vmagent` clustering	2021-02-28 21:59:55 +02:00
Aliaksandr Valialkin	3b1c2f03c3	app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration This should help debugging slow queries.	2021-02-28 19:42:29 +02:00
Aliaksandr Valialkin	0165063362	docs/CHANGELOG.md: mention about https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1074	2021-02-28 19:32:21 +02:00
Aliaksandr Valialkin	9e644ef111	lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls This should fix the `http: superfluous response.WriteHeader call` issue See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078	2021-02-28 19:23:26 +02:00
Aliaksandr Valialkin	9a2bf65134	lib/promscrape: add ability to spread scrape targets among multiple vmagent instances Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084	2021-02-28 18:40:42 +02:00
Aliaksandr Valialkin	a75ead3027	vendor: update github.com/VictoriaMetrics/metrics from v1.15.0 to v1.15.1 This can help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1096	2021-02-28 17:43:25 +02:00
Aliaksandr Valialkin	3e44d9947e	lib/promscrape/discovery/kubernetes: properly account the number of objects when watcher is stopped A follow-up for `b21b110b7a`	2021-02-28 17:06:49 +02:00
Aliaksandr Valialkin	0ef7a94056	lib/promscrape/discovery/kubernetes: add `vm_promscrape_discovery_kubernetes_*` metrics for monitoring internal state of k8s service discovery	2021-02-28 16:58:45 +02:00
Aliaksandr Valialkin	f52bdbe2a3	lib/promscrape/discovery/kubernetes: remove `resourceVersionMatch=NotOlderThan` query arg when watching for k8s object changes, since it cannot be used when `watch=1` query arg is passed	2021-02-28 16:08:43 +02:00
Aliaksandr Valialkin	6f8866a40a	lib/promscrape: fix possible deadlock in parallel execution of target relabeling	2021-02-28 16:05:32 +02:00
Aliaksandr Valialkin	5c9e657808	lib/promscrape/discovery/kubernetes: fix deadlock in startWatcherForURL reloadObjects must be called without holding aw.mu lock	2021-02-28 15:25:33 +02:00
Aliaksandr Valialkin	e77f2f8630	lib/promscrape/discovery/kubernetes: typo fix after `241ffd1f3b`	2021-02-28 15:15:27 +02:00
Aliaksandr Valialkin	82441537ff	lib/promscrape/discovery/kubernetes: pre-populate labelsByKey in reloadObject()	2021-02-28 15:09:43 +02:00
Aliaksandr Valialkin	d9c7c71abc	vendor: `make vendor-update`	2021-02-28 14:51:57 +02:00
Nikolay	b52d1e4f19	adds query params for vmalert (#1094 ) remoteWrite.url now accepts query params at provided url https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1087	2021-02-28 14:12:34 +02:00
Aliaksandr Valialkin	e003453941	lib/promscrape/discovery/kubernetes: compare sorted sets of labels in tests This should deflake tests where the order of labels isn't stable	2021-02-28 14:12:32 +02:00
Aliaksandr Valialkin	64b57c3ed3	docs/CHANGELOG.md: mention `317b0cbed2`	2021-02-28 14:03:00 +02:00
Aliaksandr Valialkin	6a21ef87b7	lib/promscrape: add missing `startWatchersForRole()` call at the beginning of `apiWatcher.getLabelsForRole`	2021-02-28 14:00:00 +02:00
Aliaksandr Valialkin	3721c5353a	app/vmbackupmanager: add docs; the vmbackupmanager is available as a part of enterprise subscription	2021-02-27 02:18:33 +02:00
Aliaksandr Valialkin	6d0e7fb8b0	lib/promscrape/discovery/kubernetes: reload k8s resources on every error This is needed for obtaining fresh resourceVersion	2021-02-27 01:46:59 +02:00
Aliaksandr Valialkin	7f1302688f	lib/fs: follow-up after `f3a03c4164`	2021-02-27 01:09:37 +02:00
Nikolay	d88fa5ebe4	Adds windows build (#1040 ) * fixes windows compilation, adds signal impl for windows, adds free space usage for windows, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036 NOTE victoria metrics database still CANNOT work under windows system, only vmagent is supported. To completly port victoria metrics, you have to fix issues with separators, parsing and posix file removall * rollback separator * Adds windows setInformation api, it must behave like unix, need to test it. changes procutil * check for invlaid param * Fixes posix delete semantic * refactored a bit * fixes openbsd build * removed windows api call * Fixes code after windows add * Update lib/procutil/signal_windows.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-27 01:06:22 +02:00
Aliaksandr Valialkin	095e61a37f	docs/CHANGELOG.md: mentioned a bugfix with `extra_label` handling during caching query results Related to `186c078fac`	2021-02-27 00:23:39 +02:00
Nikolay	673b10dd7f	adds enforced tag filters into cache key (#1095 )	2021-02-27 00:23:38 +02:00
Aliaksandr Valialkin	44975e28fe	lib/promscrape: yet another typo fix after `ed8441ec52`	2021-02-26 23:36:34 +02:00
Aliaksandr Valialkin	c1b8729bd8	lib/fs: properly handle `stale NFS file handle` error during file deletion This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.	2021-02-26 23:24:46 +02:00
Aliaksandr Valialkin	50e74c439c	lib/promscrape: typo fix after `ed8441ec52`	2021-02-26 23:04:40 +02:00
Aliaksandr Valialkin	e38ca28d99	app/vmselect/promql: increase accuracy for `buckets_limit()` function for small limits by skipping the first and the last buckets during merge The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets, then the resulting accuracy can suffer.	2021-02-26 22:56:04 +02:00
Aliaksandr Valialkin	fa3ce450fb	lib/promscrape: cache ScrapeWork This should reduce the time needed for updating big number of scrape targets.	2021-02-26 21:43:41 +02:00
Aliaksandr Valialkin	efcdf613c2	lib/promscrape/discovery/kubernetes: cache target labels This should reduce CPU usage on repeated SDConfig.GetLabels() calls.	2021-02-26 20:24:29 +02:00
Aliaksandr Valialkin	22822feea3	lib/promscrape/discovery/kubernetes: errcheck fix	2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin	dc8c045378	lib/promscrape: cleanup after `9b2246c29b` Main points: * Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit * Minimize changes in lib/promscrape/discovery/kubernetes compared to `a93e644001` * Corner case fixes.	2021-02-26 19:09:12 +02:00
Nikolay	cf9262b01f	vmagent kubernetes watch stream discovery. (#1082 ) * started work on sd for k8s * continue work on watch sd * fixes * continue work * continue work on sd k8s * disable gzip * fixes typos * log errror * minor fix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-26 19:09:12 +02:00
dereksfoster99	5b5254793d	Alphabetized names and added "native speaker" spin. (#1093 ) Do we have the dates that each of these were written so that can be added next to each brand?	2021-02-26 16:41:26 +02:00
dereksfoster99	1f088eb571	Native speaker edits. (#1088 ) I made an effort to not change anything substantive.	2021-02-26 16:41:24 +02:00
Aliaksandr Valialkin	4cfac70cde	lib/promscrape: remove duplicate code a bit	2021-02-26 15:54:07 +02:00
Aliaksandr Valialkin	55eb983193	lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel	2021-02-26 12:48:48 +02:00
Aliaksandr Valialkin	f8baf1a76d	lib/promrelabel: optimize `labeldrop` and `labelkeep` relabeling for `prefix.*` and `prefix.+` regexps	2021-02-24 17:58:11 +02:00
Aliaksandr Valialkin	0144b164c7	app/vmselect: add `sign(q)` and `clamp(q, min, max)` functions, which will be added in the upcoming Prometheus release See https://twitter.com/roidelapluie/status/1363428376162295811 The `last_over_time(m[d])` function already exists in MetricsQL.	2021-02-24 17:26:27 +02:00
Aliaksandr Valialkin	0c76828ba6	docs/CHANGELOG.md: mention about a bugfix from `4805b80977`	2021-02-24 11:48:54 +02:00
Aliaksandr Valialkin	1b90a091cf	app/vmselect/netstorage: cleanup after `4805b80977`	2021-02-24 11:43:09 +02:00
weng zhao	4805b80977	vmselect: fix vmselect cache partial result when timeout (#1085 )	2021-02-24 11:39:56 +02:00
Aliaksandr Valialkin	d16effc29e	app/vmselect/promql: properly calculate `histogram_quantile() over zero buckets and only a single non-zero` le="+Inf"` bucket like Prometheus does	2021-02-24 00:41:30 +02:00
Aliaksandr Valialkin	06676c8feb	lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache	2021-02-23 15:50:08 +02:00
faceair	b1409a7413	lib/storage: correct tagfilter match cost (#1079 )	2021-02-22 21:54:37 +02:00
Aliaksandr Valialkin	5dbe88a1c6	app/vmselect/promql: add `increase_pure()` function to MetricsQL Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962	2021-02-22 19:15:31 +02:00
Aliaksandr Valialkin	07effa77d9	app/vmselect/graphite: support `Storage-Step` header value and `storage_step` query arg at /render API	2021-02-22 18:33:51 +02:00
Aliaksandr Valialkin	197ecca426	lib/promrelabel: add more optimizations for relabeling for common cases	2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin	63c16c3fdf	lib/promrelabel: optimize relabeling performance for common cases	2021-02-22 00:51:07 +02:00
Aliaksandr Valialkin	8b87398333	lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric	2021-02-21 23:21:35 +02:00
Aliaksandr Valialkin	1adbbe7617	vendor: update github.com/VictoriaMetrics/metrics from v1.14.0 to v1.15.0 The v1.15.0 exports the following additional metrics: process_io_read_bytes_total - the number of bytes read via io syscalls such as read and pread process_io_written_bytes_total - the number of bytes written via io syscalls such as write and pwrite process_io_read_syscalls_total - the number of read syscalls such as read and pread process_io_write_syscalls_total - the number of write syscalls such as write and pwrite process_io_storage_read_bytes_total - the number of bytes read from storage layer process_io_storage_written_bytes_total - the number of bytes written to storage layer These metrics can be used for monitoring process io	2021-02-21 22:53:54 +02:00
Aliaksandr Valialkin	587132555f	lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now, so GC doesn't need visiting them.	2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin	bd3bcdc43c	lib/storage: do not re-calculate stats for heavy tag filters This should reduce the number of slow queries when stats for heavy tag filters was recalculated.	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	b8a5ee2e93	lib/{mergeset,storage}: allow merging smaller number of small parts While this may increase CPU and disk IO usage needed for background merge, this also recudes CPU usage during queries in production. This is because such queries tend to read recently added data and it is better to have lower number of parts for such data in order to reduce CPU usage. This partially reverts `ebf8da3730`	2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin	34195218e1	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:43:37 +02:00
Roman Khavronenko	14323af8c9	Add `Labels limit exceeded` panel to dashboard (#1076 ) New panel supposed to display events when VM drops extra label on exceeding `maxLabelsPerTimeseries` limit.	2021-02-19 00:50:21 +02:00
Aliaksandr Valialkin	f09b46b2b1	lib/promscrape: typo fix after the commit `f26162ec99`	2021-02-19 00:34:18 +02:00
Aliaksandr Valialkin	72eef964d9	app/vmagent: properly perform graceful shutdown, which was broken in the commit `1d1ba889fe` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065	2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin	41d3ff4f2b	app/vmagent: fix scrape config example for scrape_align_interval option	2021-02-18 23:57:45 +02:00
Aliaksandr Valialkin	502d0e2524	lib/promscrape: add scrape_align_interval config option into scrape config This option allows aligning scrapes to a particular intervals.	2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin	bd1d906eee	app/vmselect/promql: reduce the probability of `duplicate time series` errors when querying Kubernetes metrics	2021-02-18 22:08:13 +02:00
Aliaksandr Valialkin	3a8b2eed58	docs/CHANGELOG.md: cut v1.54.1	2021-02-18 19:10:57 +02:00
Aliaksandr Valialkin	10ccb92e4d	lib/storage: use composite index for a query with a name filter and negative filters	2021-02-18 18:57:45 +02:00
Aliaksandr Valialkin	418de71509	lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters Example: `node_cpu_seconds_total{mode!="idle"}`	2021-02-18 18:33:05 +02:00
Aliaksandr Valialkin	902a4f6486	app/victoria-metrics: remove accidentally added single-node tests	2021-02-18 18:33:03 +02:00
Aliaksandr Valialkin	0af9fbe326	docs/Articles.md: add a link to https://www.percona.com/blog/2021/02/12/tame-kubernetes-costs-with-percona-monitoring-and-management-and-prometheus-operator/	2021-02-18 16:37:22 +02:00
Aliaksandr Valialkin	901710b9e2	app/vmalert: add missing multiarch Dockerfile	2021-02-18 15:23:57 +02:00
Aliaksandr Valialkin	be93e02085	docs/CHANGELOG.md: cut v1.54.0	2021-02-18 14:52:52 +02:00
Aliaksandr Valialkin	628b8eb55e	lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters	2021-02-18 14:01:18 +02:00
Aliaksandr Valialkin	fd41f070db	lib/storage: sort tag filters by the number of loops they need for the execution This metric should work better than the filter execution duration, since it cannot be distorted by concurrently running queries.	2021-02-18 12:52:29 +02:00
Aliaksandr Valialkin	a5bfd831fb	docs/Articles.md: add a link to https://medium.com/alteos-tech-blog/observability-availability-and-doras-research-program-85deb6680e78	2021-02-18 01:25:54 +02:00
Aliaksandr Valialkin	1dad7ecf54	app/victoria-metrics/testdata: add a test for {__graphite__="foo.*.bar"} selector	2021-02-17 21:53:24 +02:00
Aliaksandr Valialkin	9c81429299	app/vmagent/remotewrite: cleanup after `1d1ba889fe` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065	2021-02-17 21:43:48 +02:00
Nikolay	40973eda1c	adds pushback for fastqueue, (#1075 ) during shutdown currently sending block was lost, now its pushed back to fast queue and will be flushed on disk, it may lead to data duplication. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065	2021-02-17 21:43:46 +02:00
Aliaksandr Valialkin	57d192a2c2	docs/CHANGELOG.md: mention that prod binaries are built now with Go1.16 This is a follow-up for `4edfe76bef`	2021-02-17 21:07:48 +02:00
Aliaksandr Valialkin	80699405aa	docs/CaseStudies.md: actualize Wix numbers	2021-02-17 21:03:42 +02:00
Aliaksandr Valialkin	9566015a36	Revert "lib/mergeset: tune lifetime for entries inside block caches" This reverts commit `458c89324d`. Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.	2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin	b4c7d5992b	lib/storage: move composite filters to the top during sorting	2021-02-17 20:26:32 +02:00
Aliaksandr Valialkin	2005d8212a	lib/storage: return back filter arg to getMetricIDsForTagFilter function The filter arg has been removed in the commit `c7ee2fabb8` because it was preventing from caching the number of matching time series per each tf. Now the cache contains duration for tf execution, so the filter shouldn't break such caching.	2021-02-17 19:33:15 +02:00
Aliaksandr Valialkin	83da939947	app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics	2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin	0c5bb2a397	lib/storage: revert `ecf132933e`, since negative filters require the same amount of work as non-negative filters	2021-02-17 18:56:13 +02:00
Aliaksandr Valialkin	bbc287ea6a	lib/storage: tag filters sorting...	2021-02-17 18:56:12 +02:00
Aliaksandr Valialkin	f5e841c1e9	lib/storage: further tune tag filters sorting	2021-02-17 17:28:36 +02:00
Aliaksandr Valialkin	45e5cdb631	deployment/dm: update Go builder image from v1.15.8 to v1.16.0 See release notes for Go1.16 at https://golang.org/doc/go1.16	2021-02-17 15:18:45 +02:00
Aliaksandr Valialkin	9d1f14d94c	lib/storage: tune the logic for sorting tag filters according the their exeuction times	2021-02-17 15:02:19 +02:00
Aliaksandr Valialkin	1a19702d92	lib/storage: make sure that nobody uses partitions when closing the table	2021-02-17 15:02:18 +02:00
Aliaksandr Valialkin	31e04e0d45	docs/CHANGELOG.md: document new per-tenant metrics	2021-02-16 23:33:59 +02:00
Aliaksandr Valialkin	3062ff0fdb	app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration The metrics are: - vm_vmselect_http_requests_total{accountID="...",projectID="..."} - the total number of select requests per each tenant - vm_vmselect_http_duration_ms_total{accountID="...",projectID="..."} - the total duration in milliseconds for per-tenant select requests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932	2021-02-16 23:30:29 +02:00
Aliaksandr Valialkin	2d5587d65c	vendor: `make vendor-update`	2021-02-16 22:25:02 +02:00
Aliaksandr Valialkin	3b8596bfec	lib/httpserver: make errcheck happy	2021-02-16 22:05:59 +02:00
Aliaksandr Valialkin	667a7594b7	docs: rename vmbackuper to vmbackupmanager	2021-02-16 22:01:21 +02:00
Aliaksandr Valialkin	1ab7a8dfd5	lib/storage: more tuning for tag filters sorting according the time they take	2021-02-16 21:27:27 +02:00
Aliaksandr Valialkin	35a23234ca	lib/mergeset: tune lifetime for entries inside block caches This should reduce memory usage in general case without significant CPU usage increase	2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin	500acb958c	lib/mergeset: clarify comments in the code a bit	2021-02-16 18:03:11 +02:00
Aliaksandr Valialkin	7dee57da03	deployment/docker: properly publish `latest` tag during `make publish-via-docker` This has been broken in `f9902b3372`	2021-02-16 17:45:12 +02:00
Aliaksandr Valialkin	73c9da16b8	app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations	2021-02-16 16:09:00 +02:00
Aliaksandr Valialkin	a3a09a3c6e	app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key This should prevent from a memory allocation and a string copy.	2021-02-16 15:46:31 +02:00
Aliaksandr Valialkin	da05c10544	lib/uint64set: remove memory allocation in bucket16.appendTo when sorting smallPool	2021-02-16 15:31:59 +02:00
Aliaksandr Valialkin	18305caadb	lib/httpserver: cache `/metrics` output for a second This should reduce CPU load when `/metrics` output is scraped with a frequency exceeding a request per second	2021-02-16 14:57:13 +02:00
Aliaksandr Valialkin	7869d38043	lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names	2021-02-16 13:59:11 +02:00
Aliaksandr Valialkin	d788876a7a	lib/mergeset: remove unused code after `a4140de9e6`	2021-02-16 13:40:29 +02:00
Aliaksandr Valialkin	55952f8f2e	lib/storage: tune sorting for tag filters	2021-02-16 13:07:42 +02:00
Aliaksandr Valialkin	3eae03a337	lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs	2021-02-15 16:39:52 +02:00
Aliaksandr Valialkin	46e98ed490	vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 The new version switches from log-linear histograms to log-based histograms, which provide up to 3.6 times better accuracy.	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	93ff866e91	lib/storage: reduce the minimum supported retention for inverted index from one month to one day	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	5d5d310dcc	lib/flagutil: prevent from integer overflow when parsing duration	2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin	fccb481de2	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_endpoints_label_` and `__meta_kuberntes_endpoints_annotation_` labels to `role: endpoints` This syncs kubernetes SD with Prometheus 2.25 See `617c56f55a`	2021-02-15 02:51:36 +02:00
Aliaksandr Valialkin	cb6eba2ce0	docs/Cluster-VictoriaMetrics.md: clarify replication docs	2021-02-15 01:44:22 +02:00
Aliaksandr Valialkin	54a09de037	lib/logger: explicitly import "time/tzdata" package for embedding tzdata into the app The approach with `timetzdata` build tag didn't work for GOARCH=arm and GOARCH=ppc64le due to the issue https://github.com/golang/go/issues/44073#issuecomment-778854298	2021-02-15 01:00:30 +02:00
Aliaksandr Valialkin	6f3bbf21b8	lib/storage: sort tag filters by actual execution time instead of by the number of matching time series This should improve query speed for queries with regexp filters matching small number of time series on a label with big number of unique values.	2021-02-15 00:19:46 +02:00
Aliaksandr Valialkin	9e3993c585	lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` . This could result in missing time series for such tag filters.	2021-02-14 23:39:19 +02:00
Aliaksandr Valialkin	f85c2f052f	docs/CHANGELOG.md: mention about fixed multiarch build for Docker images Related commit: `f9902b3372`	2021-02-12 15:24:16 +02:00
Aliaksandr Valialkin	18c2075159	lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes. * It is better using `up == 0` query for determining failing targets. * It is better using the following query for determining targets with exceeded limit on the number of metrics: scrape_samples_scraped > 0 if up == 0	2021-02-12 05:23:27 +02:00
Aliaksandr Valialkin	c769f8321d	deployment/docker: embed tzdata into prod Go app instead of installing it into base docker image While this increases app size by 700Kb, this allows using -loggerTimezone in a scratch base image See https://github.com/golang/go/issues/38017	2021-02-12 04:56:27 +02:00
Aliaksandr Valialkin	ff7850aec0	deployment/docker: use `docker buildx` for creating multiarch builds See https://github.com/docker/buildx/	2021-02-12 04:35:35 +02:00
Aliaksandr Valialkin	a463cda759	docs/Cluster-VictoriaMetrics.md: mention that `/api/v1/import/prometheus` supports OpenMetrics data	2021-02-12 00:59:27 +02:00
Aliaksandr Valialkin	670cc20b71	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics support data ingestion in OpenMetrics format	2021-02-12 00:57:33 +02:00
Aliaksandr Valialkin	ee4288987b	app/vmselect/prometheus: treat `match` query arg in the same way as `match[]` query arg	2021-02-11 15:01:55 +02:00
Aliaksandr Valialkin	9e88ff3075	app/vmauth: add ability to route requests from a single users to multiple targets depending on the requested path Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1064	2021-02-11 12:41:38 +02:00
Aliaksandr Valialkin	54d2f67924	app/vmselect/promql: properly make copies of EvalConfig	2021-02-11 12:41:37 +02:00
Aliaksandr Valialkin	5973fd4067	app/vmselect/promql: make a copy of EvalConfig when executing `q1` and `q2` in parallel for `q1 binary_op q2` This should prevent from data races if the underlying functions modify EvalConfig contents.	2021-02-10 23:06:58 +02:00
Aliaksandr Valialkin	985c3e301d	app/vmselect: parallelize `q1 <binary_op> q2` queries by running `q1` and `q2` in parallel This should reduce query execution times.	2021-02-10 23:00:56 +02:00
Aliaksandr Valialkin	4e645a5fd3	lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case	2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin	eeb92eb7fc	lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata	2021-02-10 17:55:51 +02:00
Aliaksandr Valialkin	08f21d8761	app/vmstorage: export vm_composite_index_min_timestamp metric	2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin	b27288f1b0	lib/storage: parallelize tag filters execution a bit This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.	2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin	4262c2f7c2	lib/storage: remove filter arg from getMetricIDsForDateTagFilter function The `filter` arg breaks the logic for sorting tag filters by the matching metrics, which may result in non-optimal performance during time series search.	2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin	681dfb7485	lib/storage: fix inconsistencies in error logs	2021-02-10 16:32:21 +02:00
Aliaksandr Valialkin	148422bcba	lib/storage: disable composite index usage when querying old data	2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin	17d5a03f6e	lib/storage: fix metric name match for composite filter	2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin	fa0ef143b1	lib/storage: optimize search by label filters matching big number of time series	2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin	5c9715a89a	lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day This should help systems with multiple CPU cores	2021-02-10 00:04:19 +02:00
Aliaksandr Valialkin	082eabf51e	lib/fs: remove the code for tracking whether the given memory region is in page cache This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.	2021-02-09 16:51:11 +02:00
Aliaksandr Valialkin	afa9cf9c57	lib/mergeset: remove dead code left after `a4140de9e6`	2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin	9ed7789fef	optimize Storage.updatePerDateData()	2021-02-09 02:59:53 +02:00
Aliaksandr Valialkin	ea328b7391	lib/storage: skip deduplication when creating inmemory data blocks The deduplication will be performed later during merging such blocks.	2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin	9a5fd4f2b1	vendor: `make vendor-update`	2021-02-09 01:12:22 +02:00
Aliaksandr Valialkin	3fdd22eb30	deployment/dm: update Go builder image from v1.15.7 to v1.15.8 See https://github.com/golang/go/issues?q=milestone%3AGo1.15.8+label%3ACherryPickApproved	2021-02-09 00:58:41 +02:00
Aliaksandr Valialkin	7b7963a77f	lib/mergeset: unconditionally cache indexdb blocks Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage. This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.	2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin	e8ee9fa7fe	app/vmstorage: export missing `vm_cache_size_bytes` metrics for indexdb and data caches	2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin	62574c478a	docs/CHANGELOG.md: mention about a bugfix for timezone data from `df0cda3ab9`	2021-02-08 16:01:26 +02:00
Aliaksandr Valialkin	2b36eb3d82	lib/cgroup: follow-up after `b9bf3cbe3e`	2021-02-08 16:01:26 +02:00
Nikolay	32603f57cc	refactored cgroups limits, (#1061 ) adds tests, remove os.Exec	2021-02-08 16:01:26 +02:00
Aliaksandr Valialkin	83e7b7b643	vendor: update github.com/VictoriaMetrics/metrics from v1.13.0 to v1.13.1	2021-02-08 14:25:43 +02:00
Aliaksandr Valialkin	9d3afdc3d3	deployment/docker: bump local/base image tag from 1.1.1 to 1.1.2, so it is built with new timezone info after the commit `df0cda3ab9`	2021-02-08 14:05:48 +02:00
Nikolay	7466bfe794	adds zoneinfo to base docker image, (#1062 ) NOTE clean up local cache with docker rmi local/base:1.1.1-alpine_3.13.1-alpine_3.13.1	2021-02-08 14:05:47 +02:00
Aliaksandr Valialkin	2dbb12563b	lib/storage: optimize data ingestion in the beginning of every hour Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-08 12:04:51 +02:00
Aliaksandr Valialkin	1a6eb0c3cf	lib/logger: exit the app if unsupported timezone value has been passed to `-loggerTimezone` While at it, clarify descrption for `-loggerTimezone` command-line flag.	2021-02-07 23:33:35 +02:00
Aliaksandr Valialkin	35eb04b7dd	docs/vmctl.md: fix title, so it is properly displayed in the header of https://victoriametrics.github.io/	2021-02-04 20:20:38 +02:00
Aliaksandr Valialkin	191bc0bcf3	docs: sync with master branch	2021-02-04 20:02:24 +02:00
Aliaksandr Valialkin	c6a7288109	lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion This should reduce possible CPU usage spikes at the beginning of every hour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-04 18:46:23 +02:00
Aliaksandr Valialkin	8e5dad8483	docs/CHANGELOG.md: mention recently added changes	2021-02-04 16:42:33 +02:00
Aliaksandr Valialkin	b5eba70595	lib/httpserver: expose `process_open_fds` and `process_max_fds` metrics See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037	2021-02-04 16:42:32 +02:00
Nikolay	6fc2a8e544	fixes dockerswarm (#1053 ) fixes improper usage of host network services https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028	2021-02-04 15:57:41 +02:00
Roman Khavronenko	2aa37b0450	vmalert: mention `-datasource.appendTypePrefix` in README (#1052 )	2021-02-03 23:48:45 +02:00
Dmitry Shevchuk	007fd6ce9c	Adds ability to query right vmselect endpoint based on the query type (#1050 ) * Adds ability to query right vmselect endpoint based on the query type Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2021-02-03 23:48:44 +02:00
Aliaksandr Valialkin	e802daa9ee	docs/CHANGELOG.md: cut v1.53.1	2021-02-03 22:51:06 +02:00
Aliaksandr Valialkin	88ee836d0c	docs/vmalert.md: mention that `type` option can be set at group level additionally to rule level	2021-02-03 21:12:39 +02:00
Aliaksandr Valialkin	a3aa9bdc9f	app/vmagent: add `Advanced usage` section with the description for all the command-line flags	2021-02-03 21:03:52 +02:00
Aliaksandr Valialkin	e6bcef6514	docs/CHANGELOG.md: mention that `{__graphite__="foo.*.bar"}` syntax deprecates `-search.treatDotsAsIsInRegexps` command-line flag	2021-02-03 20:42:18 +02:00
Aliaksandr Valialkin	e4a013d5d7	docs/Single-server-VictoriaMetrics.md: remove misleading section about native format in `How to export data in JSON line format` chapter	2021-02-03 20:42:17 +02:00
Aliaksandr Valialkin	8de3a329ff	app/vmselect: deprecate `-search.treatDotsAsIsInRegexps` in favor to `{__graphite__="foo.*.bar"}` syntax	2021-02-03 20:36:42 +02:00
Aliaksandr Valialkin	8249f13104	app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards Example: foo{bar{x,yz},a[b-c],*de}	2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin	2976ec89b8	lib/storage: fix a bug, which breaks searching by Graphite wildcard filters	2021-02-03 20:15:50 +02:00
Aliaksandr Valialkin	45a63a1da9	sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks	2021-02-03 20:15:37 +02:00
Aliaksandr Valialkin	785b770af3	docs/CHANGELOG.md: fix a link to Graphite Render API usage docs	2021-02-03 12:30:09 +02:00
Aliaksandr Valialkin	844f9991be	docs: mention about Graphite render API implementation	2021-02-03 12:13:08 +02:00
Aliaksandr Valialkin	b0acea9d4f	docs/Cluster-VictoriaMetrics.md: document Graphite Render API endpoint	2021-02-03 12:03:40 +02:00
Aliaksandr Valialkin	c69b52cf31	docs/CHANGELOG.md: cut v1.53.0	2021-02-03 03:43:14 +02:00
Aliaksandr Valialkin	71a9d4ecd3	deployment/docker: update base alpine image from v3.13.0 to v3.13.1 See release notes for v3.13.1 - https://www.alpinelinux.org/posts/Alpine-3.13.1-released.html	2021-02-03 03:40:20 +02:00
Aliaksandr Valialkin	52b218c07e	docs/Single-server-VictoriaMetrics.md: sync with master branch	2021-02-03 01:37:55 +02:00
Aliaksandr Valialkin	4b930b9ffe	app/vmselect: add ability to set Graphite-compatible filter via `{__graphite__="foo.*.bar"}` syntax	2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin	755b0998ce	lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric	2021-02-02 16:16:51 +02:00
Aliaksandr Valialkin	4c59dbc127	lib/promscrape: add `vm_promscrape_scrape_retries_total`, `vm_promscrape_discovery_retries_total` and `vm_promscrape_discovery_requests_total` metrics	2021-02-01 20:06:16 +02:00
Aliaksandr Valialkin	03da0b728c	docs: increase heading sizes in vmagent, vmauth, vmbackup and vmrestore docs, so they match the heading sizes in VictoriaMetrics docs	2021-02-01 19:44:58 +02:00
Aliaksandr Valialkin	f4a4665857	docs/vmctl.md: update build instructions after the migration from github.com/VictoriaMetrics/vmctl to github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl	2021-02-01 19:44:57 +02:00
Aliaksandr Valialkin	e582eede8a	vendor: make vendor-update	2021-02-01 19:32:36 +02:00
Aliaksandr Valialkin	286625e4b3	docs: sync with master branch	2021-02-01 18:05:58 +02:00
Aliaksandr Valialkin	7d23f3ff3a	app/vmselect: add ability to set an additional label filters via `extra_label` query arg	2021-02-01 17:42:35 +02:00
Aliaksandr Valialkin	6811445b64	docs: document ability to query Graphite datasource from vmalert	2021-02-01 15:28:31 +02:00
Nikolay	b8bc1c2e0f	Graphite vmalert wip (#112 ) * init implementation for graphite alerts * adds graphite support for vmalert * small fix * changes vmalert graphite api with type * updates tests * small fix * fixes graphite parse * Fixes graphite from time	2021-02-01 15:28:30 +02:00
Aliaksandr Valialkin	e79799a784	lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func	2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin	fdf9de98f8	app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values This commit also adds --vm-round-digits command-line option to vmctl tool.	2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin	05474aaa29	app/vmctl: fix `make check-all` warnings	2021-02-01 01:31:45 +02:00
Aliaksandr Valialkin	7f4fb34182	app/vmctl: move vmctl code from github.com/VictoriaMetrics/vmctl It is better developing vmctl tool in VictoriaMetrics repository, so it could be released together with the rest of vmutils tools such as vmalert, vmagent, vmbackup, vmrestore and vmauth.	2021-02-01 01:18:39 +02:00
Aliaksandr Valialkin	5328a102e0	app/vmselect: unconditionally deny partial responses from `/api/v1/export` It is expected that `/api/v1/export` returns full data, so there is no sense in partial responses there.	2021-01-27 14:39:53 +02:00
Aliaksandr Valialkin	ff526492eb	docs/Cluster-VictoriaMetrics.md: mention about `-search.denyPartialResponse` command-line flag and `deny_partial_response` query arg	2021-01-27 14:06:14 +02:00
Aliaksandr Valialkin	db5b78f65c	docs/CHANGELOG.md: typo fixes	2021-01-27 01:20:31 +02:00
Aliaksandr Valialkin	2c14c70903	lib/logger: initialize timezone by UTC in order to fix failing tests	2021-01-27 01:00:57 +02:00
Aliaksandr Valialkin	ffec5131ae	lib/promscrape: export `vm_promscrape_scrapes_failed_per_url_total` and `vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total` metrics These metrics could be useful for determining imporperly working scrape targets. Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	4b324da947	all: consistently use timers from timerpool	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	588090765c	lib/fs: properly initialize cleaner for pageCache bitmaps Previously it wasnt working because the timer was fired only once	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	29bf531f7d	app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035	2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin	68a66be811	lib/logger: add `-loggerTimezone` command-line flag for adjusting timezone for timestamps in log messages	2021-01-26 22:53:25 +02:00
Aliaksandr Valialkin	9e175683b1	docs/CHANGELOG.md: mention about https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1027	2021-01-26 16:38:22 +02:00
weng zhao	2a8a34ea05	vmalert: add option datasource.queryStep to allow user to address the inconsistency between grafana dashboards(query_range with step 15s usually) and ALERTS (#1027 ) Co-authored-by: zhao.weng <zhao.weng@shopee.com>	2021-01-26 16:38:20 +02:00
Aliaksandr Valialkin	fdced59278	lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection	2021-01-22 13:22:59 +02:00
Aliaksandr Valialkin	44c74f1e79	app/vmselect/promql: improve documentation for `-search.maxPointsPertimeseries` command-line flag This should reduce incorrect usage and assumptions for this flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020	2021-01-22 13:00:35 +02:00
Aliaksandr Valialkin	7308691865	docs/CHANGELOG.md: mention about the fix with too big HTTP reconnection rate to targets This has been fixed in `0a45220b0a`	2021-01-22 12:10:01 +02:00
Aliaksandr Valialkin	b0386234b7	deployment/docker: update Go builder from v1.15.6 to v1.15.7 See https://groups.google.com/g/golang-nuts/c/ufLjEY_AJ0I/m/smSHpGXiDQAJ for details	2021-01-21 18:40:23 +02:00
Aliaksandr Valialkin	6a13d72596	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.11 to v1.0.12	2021-01-21 12:00:17 +02:00
Aliaksandr Valialkin	e55205220b	app/vmselect: add `-search.maxStepForPointsAdjustment` command-line flag, which can be used for disabling adjustment for points returned from `/api/v1/query_range` handler if they have timestamps closer than `-search.latencyOffset` to the current time	2021-01-19 22:57:50 +02:00
Aliaksandr Valialkin	5856611291	app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()	2021-01-18 17:31:27 +02:00
Aliaksandr Valialkin	79cd33319f	LICENSE: bump the last year from 2020 to 2021	2021-01-16 13:00:55 +02:00
Aliaksandr Valialkin	100505b33d	deployment/docker: upgrade alpine base Docker image from v3.12.3 to v3.13.0 See release notes for v3.13.0 - https://www.alpinelinux.org/posts/Alpine-3.13.0-released.html	2021-01-15 22:50:24 +02:00
faceair	cf76d8dd79	lib/mergeset: add missing shouldCacheBlock (#1019 )	2021-01-15 11:47:25 +02:00
Aliaksandr Valialkin	5640e6cbca	docs/vmagent.md: follow-up for `184a659c5f`	2021-01-13 13:54:28 +02:00
mancubus77	6f3560d256	Doco vmagent fix (#1014 ) * Update section with remote_write.url for clustered version * fix typo Co-authored-by: mancubus <dont@write.me>	2021-01-13 13:54:27 +02:00
Aliaksandr Valialkin	d23d6d4bfa	docs/CHANGELOG.md: cut v1.52.0	2021-01-13 12:59:08 +02:00
Aliaksandr Valialkin	c5bdab5a4c	app/vmselect/promql: add ability to pass multiple labels to `sort_by_label` and `sort_by_label_desc` functions See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992	2021-01-13 12:43:47 +02:00
Aliaksandr Valialkin	253de4e827	vendor: `make vendor-update`	2021-01-13 12:19:53 +02:00
Aliaksandr Valialkin	81789da731	lib/backup: increase backup chunk size from 128MB to 1GB This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing	2021-01-13 12:16:39 +02:00
Aliaksandr Valialkin	8cae98aa78	app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990	2021-01-13 10:59:32 +02:00
Aliaksandr Valialkin	2d4006cb4a	docs/CHANGELOG.md: document updated `extra_label` query arg behavior Follow-up for `dc9d7aedd5`	2021-01-13 01:07:25 +02:00
Nikolay	821492bc0b	adds extra_label to all import apis (#1007 ) * adds extra_label to all import apis, changes priority for extra_label - now it has priority over original labels * Update README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * Update README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds extra labels to vmagent import api changes order for adding labels, now its added after user values * adds tests for extra_label * import fix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-01-13 01:07:24 +02:00
Aliaksandr Valialkin	bc8b38daca	docs/CHANGELOG.md: mention that the minimum supported TLS version now is v1.2 Follow-up for `7bf5d48315`	2021-01-13 00:45:08 +02:00
Nikolay	095be83f2f	bumps minimal tls version (#1012 )	2021-01-13 00:39:09 +02:00
Aliaksandr Valialkin	b71748f1d9	docs/Single-server-VictoriaMetrics.md: typo fix	2021-01-12 22:03:46 +02:00
Aliaksandr Valialkin	8711860327	deployment/docker: upgrade base image for Docker packages from Alpine 3.13.1 to Alpine 3.12.3 in order to fix potential security issues See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1010	2021-01-12 21:57:07 +02:00
Aliaksandr Valialkin	df6e399f73	app/vmselect/promql: add `tfirst_over_time(m[d])` and `tlast_over_time(m[d])` MetricsQL functions for returning timestamps for the first and the last samples in `m` over `d`	2021-01-12 16:12:47 +02:00
Aliaksandr Valialkin	de04dd81cf	docs/Articles.md: add https://cer6erus.medium.com/cloud-native-model-driven-telemetry-stack-on-openshift-80712621f5bc	2021-01-12 15:36:50 +02:00
Aliaksandr Valialkin	3d79471fb3	lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here This is a follow-up commit after `c8ea697db8`	2021-01-12 15:20:22 +02:00
Aliaksandr Valialkin	719ad49adf	lib/storage: de-duplicate tags in MetricName.sortTags Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.	2021-01-12 15:03:22 +02:00
Aliaksandr Valialkin	1dee98a331	docs/CHANGELOG.md: document big fixes from the commit `7976c22797`	2021-01-12 13:44:10 +02:00
Nikolay	9f0a4fd00e	Fixes error handling for promscrape.streamParse (#1009 ) properly return error if client cannot read data, properly suppress scraper errors	2021-01-12 13:35:09 +02:00
Aliaksandr Valialkin	c97681b45c	lib/promscrape: properly show scrape duration on `/targets` page Previously it has been shown as 0.000s for any scrape duration.	2021-01-11 21:15:50 +02:00
Aliaksandr Valialkin	bdf67a13cc	docs/Single-server-VictoriaMetrics.md: mention about https://github.com/aorfanos/vmalert-cli in `Integrations` section	2021-01-11 18:52:56 +02:00
Aliaksandr Valialkin	4731780a38	docs/CHANGELOG.md: mention about a bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989	2021-01-11 13:11:53 +02:00
Roman Khavronenko	304512b668	vmalert-989: return non-empty result in template func `query` stub to pass validation (#1002 ) On templates validation stage vmalert does not acutally send queries, so for complex chained expression validation may fail. To avoid this, we add a blank sample in response so validation can pass successfully. Later, during the rule execution, stub will be replaced with real `query` function. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989	2021-01-11 12:59:33 +02:00
Aliaksandr Valialkin	4ee53c3961	all: use `net.Dial` instead of `fasthttp.Dial`, because `fasthttp.Dial` limits the number of concurrent dials to 1000	2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin	53e3df9449	vendor: update github.com/VictoriaMetrics/fasthttp	2021-01-11 12:52:49 +02:00
Aliaksandr Valialkin	d5a2b120e9	app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage	2021-01-08 00:12:12 +02:00
Aliaksandr Valialkin	bd7cb4f5be	vendor: `make vendor-update`	2021-01-08 00:12:10 +02:00
Aliaksandr Valialkin	fa176dacf1	docs/Single-server-VictoriaMetrics.md: sync with upstream	2021-01-07 23:39:21 +02:00
Aliaksandr Valialkin	5fb8cf00c3	docs/Cluster-VictoriaMetrics.md: typo fix	2021-01-07 23:39:12 +02:00
Nikolay	137efe6c71	Snap docs change (#986 ) * adds snap docs, adds release information for snap package, adds docs notes about configuration management with snap package. * adds release page mention * version fix for snap, its awful * revert version * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-29 11:44:50 +02:00
Aliaksandr Valialkin	e2d8b9e091	README.md: mention about `-search.queryStats.lastQueriesCount` and `-search.queryStats.minQueryDuration` command-line flags in docs about query stats	2020-12-29 11:39:45 +02:00
Aliaksandr Valialkin	66a19e0079	docs/CHANGELOG.md: cut v1.51.0	2020-12-27 14:21:41 +02:00
Aliaksandr Valialkin	47872ada7e	app/vmselect/promql: do not ajdust `offset` value provided in the query Previously it could be modified in order to improve response cache hit ratio. This is unneeded, since cache hit ratio should remain good because the query time range should be already aligned to multiple of `step` values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976	2020-12-27 14:10:15 +02:00
Aliaksandr Valialkin	5c8e4cf03e	docs/Articles.md: add a link to https://www.percona.com/blog/2020/12/23/observations-on-better-resource-usage-with-percona-monitoring-and-management-v2-12-0/	2020-12-27 13:02:08 +02:00
Aliaksandr Valialkin	5bbf200de2	app/vmselect: add per-tenant `/api/v1/status/top_queries` handler	2020-12-27 12:53:50 +02:00
Aliaksandr Valialkin	0e739efc88	app/vmselect/promql: simplify defer call for querystats.RegisterQuery	2020-12-27 12:07:56 +02:00
Aliaksandr Valialkin	44932098b5	app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K This should reduce memory usage in constrained environments	2020-12-25 17:40:32 +02:00
Aliaksandr Valialkin	e6deb39064	app/vmselect: refactor `/api/v1/stats/top_queries` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907	2020-12-25 17:24:25 +02:00
Nikolay	76d092c091	Adds query stats handler (#945 ) * Adds query stat handler, for query and query_range api, victoriametrics tracks query execution time, stats are expored at /api/v1/status/queries endpoint with topN param https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907 * fixed query stats bugs * improves queryStats tracker * improves query stat * small fix * fix tests * added more tests * fixes 386 tests * naming fixes * adds drop for outdated records	2020-12-25 17:24:24 +02:00
Aliaksandr Valialkin	ca8919e8e1	lib/storage: wait for pending transactions before closing and dropping the partition This deflakes `make test-full-386` test	2020-12-25 11:46:47 +02:00
Aliaksandr Valialkin	991262d53e	docs/CHANGELOG.md: mention that vmalert now properly escapes multi-line queries when passing to Grafana A follow-up for `1de15ad490` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890	2020-12-25 11:12:57 +02:00
Nikolay	14915071d6	adds escape for CRLF (#984 ) at external.alert.source - \n and \r symbols was url encoded, instead of direct usage. replace it from "\n" to `\n` allows to skip url encoding. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890	2020-12-25 11:06:47 +02:00
Aliaksandr Valialkin	e061938181	vendor: `make vendor-update`	2020-12-24 17:17:39 +02:00
Aliaksandr Valialkin	c0511144e3	lib/storage: physically remove stale parts Previously they were removed from partition struct, but the corresponding directories weren't removed. This is a follow-up for `46dba00756`	2020-12-24 16:56:09 +02:00
Aliaksandr Valialkin	b480585905	app/vmalert: typo fix in descriptions for notifier.basicAuth.username and notifier.basicAuth.password command-line flags	2020-12-24 12:49:40 +02:00
Nikolay	dd2f815204	fixes kubernetes_sd (#983 ) * fixes kubernetes_sd, adds missing service metadata for pod ports without endpoint https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982 * fix test	2020-12-24 11:34:12 +02:00
Aliaksandr Valialkin	2a8bd2b5cc	docs/CHANGELOG.md: mention about adding missing __meta_kubernetes_service_* labels for `endpoints` and `endpointslices` roles in `kubernetes_sd_config` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982	2020-12-24 11:34:01 +02:00
Aliaksandr Valialkin	367fc17933	lib/promscrape: code prettifying for `8dd03ecf19`	2020-12-24 10:57:20 +02:00
Nikolay	b00f7816e2	adds proxy_url support, (#980 ) * adds proxy_url support, adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery, adds proxy_url to the scrape_config for targets scrapping, http based proxy is supported atm, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503 * fixes imports	2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin	66f8fbbb32	lib/storage: do not remove parts outside the configured retention if they are currently merged These parts are automatically removed after the merge is complete.	2020-12-24 09:02:12 +02:00
Nikolay	0b87f02602	fixes panic (#979 ) * fixes panic https://github.com/VictoriaMetrics/helm-charts/issues/89 * add fast-path * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-24 09:01:40 +02:00
Aliaksandr Valialkin	d8511b6651	docs: mention that it is possible to set multiple `-notifier.tlsInsecureSkipVerify` command-line flags for vmalert See c3a92968343c2b3619f1ab935702d0e9b3a46733	2020-12-22 22:32:56 +02:00
Nikolay	67e470e598	changes vmalert notifier flag, (#978 ) fixes issue with notifier insecure setting, now its possible to use multiple notifier.tlsInsecureSkipVerify multiple time.	2020-12-22 22:27:03 +02:00
Aliaksandr Valialkin	fa3bcf220f	lib/storage: remove stale parts as soon as they go outside the configured retention Previously such parts could remain undeleted for long durations until they are merged with other parts. This should help for `-retentionPeriod` values smaller than one month.	2020-12-22 19:55:07 +02:00
Aliaksandr Valialkin	675056c71e	vendor: upgrade github.com/klauspost/compress from v1.11.3 to v1.11.4	2020-12-21 08:57:24 +02:00
Artem Navoiev	f6b76b5c01	add linkedin to release announcement	2020-12-21 08:57:23 +02:00
Artem Navoiev	618ecd4708	Announcement guide	2020-12-20 00:45:02 +02:00
Aliaksandr Valialkin	eaf51ede0f	vendor: `make vendor-update`	2020-12-19 16:59:27 +02:00
Aliaksandr Valialkin	787cbd2f7e	docs/CHANGELOG.md: cut v1.50.2	2020-12-19 15:32:45 +02:00
Aliaksandr Valialkin	a5b17946fe	docs/CHANGELOG.md: mention about bugfix for populating template variables in vmalert See `404cbd1522` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974	2020-12-19 14:21:28 +02:00
Roman Khavronenko	9ce8b36d2a	vmalert-974: fix order for labels templating (#975 ) The change fixes bug caused by `3adf8c5a6f`. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974	2020-12-19 14:21:27 +02:00
Aliaksandr Valialkin	262cf81757	app/vmselect: properly parse negative combined offsets such as `-1h2m3s` Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.	2020-12-19 01:25:03 +02:00
Aliaksandr Valialkin	5aa2b0de55	docs/CHANGELOG.md: there is no visible difference for CPU usage and disk IO usage on production workloads	2020-12-19 00:51:32 +02:00
Aliaksandr Valialkin	6859737329	lib/storage: properly determine max rows for output part when merging small parts	2020-12-18 23:26:28 +02:00
Aliaksandr Valialkin	edbe35509e	lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage	2020-12-18 20:01:20 +02:00
Aliaksandr Valialkin	c5b47df8a4	docs/CHANGELOG.md: mention that Docker images for vmagent, vmalert, vmauth, vmbackup and vmrestore with tags containing `-cluster` suffix are no longer published See `441822c4cc` for details	2020-12-18 20:01:18 +02:00
Aliaksandr Valialkin	441822c4cc	Makefile: do not build vmagent, vmalert, vmauth, vmbackup and vmrestore binaries in cluster branch, since they are identical to binaries in single-node version There are subtle differences between these binaries in cluster branch from binaries in single-node branch such as missing `-httpAuth.*` command-line flags. These differences introduce confusion to users, so let's stop building these binaries from cluster version.	2020-12-18 18:29:48 +02:00
Aliaksandr Valialkin	39f7fa4743	docs: alphabetically sort links to case studies	2020-12-18 12:31:03 +02:00
Aliaksandr Valialkin	4f4710ef9b	Makefile: force running Makefile rules if there is a file matching their names	2020-12-18 12:20:26 +02:00
Aliaksandr Valialkin	5b2fd08b27	docs/Articles.md: classify articles by themes	2020-12-18 12:05:39 +02:00
Aliaksandr Valialkin	6edd3330b4	add a link to https://www.percona.com/blog/2020/12/16/percona-monitoring-and-management-migration-from-prometheus-to-victoriametrics-faq/	2020-12-18 11:44:31 +02:00
Aliaksandr Valialkin	d7be1a0124	Upgrade github.com/valyala/gozstd from v1.8.3 to v1.9.0	2020-12-17 15:13:35 +02:00
Aliaksandr Valialkin	1ee5a234dc	lib/promscrape: remove `ID` field from `ScrapeWork` struct. Use a pointer to ScrapeWork as a key in targetStatusMap This simplifies the code a bit.	2020-12-17 14:31:55 +02:00
Aliaksandr Valialkin	4fd2973e7c	lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2 Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.	2020-12-16 23:42:17 +02:00
BigFish	60dd48c9eb	lib/protoparser/prometheus/parser.go (#970 ) fix parse timestamp error if there are some whitespaces after timestamp	2020-12-16 23:42:16 +02:00
Aliaksandr Valialkin	49b6c6f6ab	docs/FAQ.md: add a link to https://valyala.medium.com/promscale-vs-victoriametrics-resource-usage-on-production-workload-91c8e3786c03 in the question about benchmarks	2020-12-16 23:15:49 +02:00
Aliaksandr Valialkin	2b5700b1c4	docs/Articles.md: add a link to https://valyala.medium.com/promscale-vs-victoriametrics-resource-usage-on-production-workload-91c8e3786c03	2020-12-16 14:27:01 +02:00
Aliaksandr Valialkin	f6b51889ce	docs/Cluster-VictoriaMetrics.md: fix incorrect commands for profile collecting	2020-12-16 01:06:49 +02:00
Aliaksandr Valialkin	69b3ca37d0	docs/Cluster-VictoriaMetrics.md: add `Profiling` section	2020-12-16 00:59:53 +02:00
Aliaksandr Valialkin	34622f7f9b	docs/CHANGELOG.md: cut v1.50.1 release	2020-12-15 21:11:53 +02:00
Aliaksandr Valialkin	1022505131	docs/Release-Guide.md: actualize release docs	2020-12-15 21:11:02 +02:00
Aliaksandr Valialkin	f89c3e505a	docs/CHANGELOG.md: mention about bugfix, which properly removes unregistered targets from `/targets` page See bugfix at `a4c7fcb5e1`	2020-12-15 21:07:28 +02:00
Aliaksandr Valialkin	c4ec977934	lib/promscrape: properly remove deleted target from `/targets` page Previously `sw` variable wasn't captured correctly by the started goroutine.	2020-12-15 20:58:37 +02:00
Aliaksandr Valialkin	49e800ba55	app/vmagent: add `vmagent_remotewrite_blocks_sent_total` and `vmagent_remotewrite_bytes_sent_total` metrics per each `-remoteWrite.url`	2020-12-15 20:41:08 +02:00
Aliaksandr Valialkin	9ab7ca1133	docs/vmagent.md: typo fix: pearsed->parsed	2020-12-15 19:03:35 +02:00
Aliaksandr Valialkin	11674a9b76	docs/vmagent.md: mention that `sample_limit` option has no sense when stream parsing is enabled	2020-12-15 18:44:19 +02:00
Aliaksandr Valialkin	dd47c8f084	docs/CHANGELOG.md: cut v1.50.0 release	2020-12-15 14:45:00 +02:00
Aliaksandr Valialkin	a66af20686	.github/workflows/main.yml: fall back to `go get` instead of `go install` for installing aux tools It is unclear why `go install` doesn't work in Github Actions. Needs additional investigation. The following error is returned now: cannot find package "golang.org/x/lint/golint" in any of: /opt/hostedtoolcache/go/1.15.5/x64/src/golang.org/x/lint/golint (from $GOROOT) /home/runner/go/src/golang.org/x/lint/golint (from $GOPATH)	2020-12-15 14:19:11 +02:00
Aliaksandr Valialkin	eddc2bd017	lib/promscrape: properly handle scrape errors when stream parsing is enabled See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967	2020-12-15 14:10:52 +02:00
Aliaksandr Valialkin	8d1031c29a	app/vmselect/promql: return expected `increase()` result for the first point on the graph with value not exceeding 100	2020-12-15 14:10:50 +02:00
Aliaksandr Valialkin	3f88e27d0f	Do not set `GO111MODULE=off` during `go install`, since this doesnt work in Go1.14 and Go1.15	2020-12-15 13:14:41 +02:00
Aliaksandr Valialkin	aca669c89c	docs/CHANGELOG.md: mention that vmagent now accepts multiple `-remoteWrite.sendTimeout` and `-remoteWrite.tlsInsecureSkipVerify` command-line flags	2020-12-15 12:59:34 +02:00
Nikolay	7064c4eb8e	adds new Array Flags (#965 ) * adds ArrayDuration and ArrayBool flags, makes sendTimeout and tlsInsecure configurable per remoteWrite url * added backward compatibility testcases for ArrayDuration and ArrayBool * fixes bool flag * fixes test cases	2020-12-15 12:59:33 +02:00
Aliaksandr Valialkin	104aac170e	lib/promscrape: add bootstrap styles to `/targets` html page	2020-12-15 12:38:29 +02:00
Aliaksandr Valialkin	ad961fe1f1	lib/promscrape: formatting fixes for `/tarets` page	2020-12-15 11:59:28 +02:00
Aliaksandr Valialkin	38145cfbb8	lib/promscrape: formatting fixes for `/targets` page	2020-12-15 11:27:22 +02:00
Aliaksandr Valialkin	e17ac90f59	.github/workflows/main.yml: set GO111MODULE=off when installing auxiliary tools via `go install`	2020-12-15 01:02:13 +02:00
Aliaksandr Valialkin	e85159813f	docs/CHANGELOG.md: mention about adding `query`, `first` and `value` functions to alert templates See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539	2020-12-14 20:17:46 +02:00
Roman Khavronenko	9f578e389c	vmalert: add function "query", "first" and "value" to alert templates functions (#960 ) The commit adds a support for template function `query`, `first` and `value`. The function `query` executes a MetricsQL query for active alerts. In vmalert we update templates on every evaluation for active alerts to keep them up to date. With `query` func it may become a perf issue since it will fire a query on every execution. We should keep it in mind for now. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539	2020-12-14 20:12:16 +02:00
Aliaksandr Valialkin	dafef21001	all: use `go install` instead of `go get` for installing auxiliary tools This is a preparation for Go 1.16, which deprecates `go get` for installing binaries. See https://tip.golang.org/doc/go1.16#go-command : go install, with or without a version suffix (as described above), is now the recommended way to build and install packages in module mode. go get should be used with the -d flag to adjust the current module's dependencies without building packages, and use of go get to build and install packages is deprecated. In a future release, the -d flag will always be enabled.	2020-12-14 20:07:20 +02:00
Aliaksandr Valialkin	09a03b862d	docs/CHANGELOG.md: mention bugfix for proper recovering from incorrectly stored persistent queue See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964	2020-12-14 19:29:23 +02:00
Aliaksandr Valialkin	d98a2f217b	lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964	2020-12-14 19:25:53 +02:00
Aliaksandr Valialkin	b904ae3722	lib/promscrape: add missing whitespace between duration and `ago` word at `/targets` page	2020-12-14 14:20:30 +02:00
Aliaksandr Valialkin	a2eb451de4	app/{vmagent,vminsert}: follow-up for `ce8c2dd1f1`: return `/targets` page in HTML when requested via web browser	2020-12-14 14:13:01 +02:00
Nikolay	324e3aa1a5	Changes targets api (#961 ) * changes /targets api adds html response if requester accepts text/html, adds quick template for /targets api, fixes pathPrefix for / requests * changes namings * renamed targets file * Update app/victoria-metrics/main.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds trimspace to qtpl, moves content-type for targets response closer to writer * fixes bug with prefix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-14 14:13:00 +02:00
Aliaksandr Valialkin	756fc6fc6c	app/victoria-metrics: automatically reset response cache when samples with too timestamps older than `now - search.cacheTimestampOffset` are ingested	2020-12-14 13:10:16 +02:00
Aliaksandr Valialkin	c0db28cd9a	docs/MetricsQL.md: clarify that `limitk(k, q)` returns an arbitrary set of `k` time series with each call Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951	2020-12-12 14:18:02 +02:00
Aliaksandr Valialkin	5a846f3e52	vendor: `make vendor-update`	2020-12-11 23:32:50 +02:00
Aliaksandr Valialkin	eed84ac2b5	docs/FAQ.md: use less confusing links in the chapter explaining why VictoriaMetrics doesnt support Prometheus remote_read API	2020-12-11 21:23:55 +02:00
Aliaksandr Valialkin	fc82c22e50	docs: consistently use links to https://victoriametrics.github.io for documentation references	2020-12-11 21:09:17 +02:00
Aliaksandr Valialkin	f1b303e70d	docs/Single-server-VictoriaMetrics.md: clarify docs in `Relabeling` section	2020-12-11 18:24:31 +02:00
Aliaksandr Valialkin	3ec5387a36	docs/CHANGELOG.md: mention https://github.com/VictoriaMetrics/VictoriaMetrics/issues/955	2020-12-11 17:49:04 +02:00
Aliaksandr Valialkin	c80d38f00c	lib/promscrape/discovery/consul: reduce load on Consul API server by increasing timeout for blocking requests from 50 seconds to 9 minutes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-11 17:26:34 +02:00
Aliaksandr Valialkin	d6f9bf2d19	app/vmselect/graphite: properly handle wildcards and charsets inside curly braces For example, `foo{bar,[a-f]ab}` should match `foobar`, `foobar123`, `foofab`, etc. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952	2020-12-11 17:26:32 +02:00
Aliaksandr Valialkin	9e79fc27c8	app/vminsert/netstorage: properly update `vm_rpc_rerouted_rows_processed_total` metric Previously this metric wasn't updated because of improper defer call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/955 Thanks to @xemxx for spotting the bug.	2020-12-11 13:07:05 +02:00
Aliaksandr Valialkin	82d26d9dfe	docs/Single-server-VictoriaMetrics.md: sync with upstream README.md via `make docs-sync`	2020-12-11 12:09:17 +02:00
faceair	054ad542b0	docs/CaseStudies.md: add case study for zhihu (#956 )	2020-12-11 12:09:16 +02:00
Aliaksandr Valialkin	4804e004f3	docs/Single-server-VictoriaMetrics.md: clarify that the recommended value for `-dedup.minScrapeInterval` is `scrape_interval` from Prometheus configs	2020-12-09 12:16:25 +02:00
Aliaksandr Valialkin	a52924c7a3	docs/CHANGELOG.md: mention about memory leak fix in vmagent when big number of targets is discovered via service discovery	2020-12-09 10:36:05 +02:00
Aliaksandr Valialkin	5b1c4f702e	docs/Single-server-VictoriaMetrics.md: sync with upstream README.md via `make docs-sync`	2020-12-09 10:28:24 +02:00
Aliaksandr Valialkin	a84467958a	lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg. See https://www.consul.io/api-docs/health#list-nodes-for-service for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170	2020-12-08 21:53:23 +02:00
Aliaksandr Valialkin	1a237c6903	all: properly handle CPU limits set on the host system/container This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin	38188e1d6b	lib/promscrape: store ScrapeWork items by pointer in the slice returned from `get*ScrapeWork()` This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they are no longer used. See the related commit `e205975716` and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-12-08 17:55:21 +02:00
Aliaksandr Valialkin	bd8eef2528	app/vmselect/promql: do not reduce lookbehind window for `any_rollup_func(m)` to `-search.maxStalenessInterval`. It should equal to `step` value passed to `/api/v1/query_range` as most users expect	2020-12-08 15:17:05 +02:00
kreedom	996ba2770b	Create CODE_OF_CONDUCT_RU.md	2020-12-08 15:16:59 +02:00
Aliaksandr Valialkin	7bdf07883b	app/{vmalert,vmagent}: skip empty values in `-remoteWrite.label` and `-label` lists	2020-12-08 14:54:02 +02:00
Aliaksandr Valialkin	d5faad0240	lib/promscrape: re-use strings for labels stored in ScrapeWork This should reduce memory usage when working with big number of scrape targets.	2020-12-08 12:23:44 +02:00
Aliaksandr Valialkin	06091cfdf8	lib/promscrape: export `vm_promscrape_scrapers_{started\|stopped}_total` metrics for monitoring target churn rate	2020-12-08 11:58:44 +02:00
Aliaksandr Valialkin	affcee199c	lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.	2020-12-08 11:52:20 +02:00
Aliaksandr Valialkin	56a0b058c1	lib/promscrape: export `vm_promscrape_active_scrapers{type="<sd_type>"}` metric for tracking the number of active scrapers per each service discovery type	2020-12-08 01:54:44 +02:00
Aliaksandr Valialkin	b5b32c65b0	lib/promscrape: do not enable strict config parsing when `-promscrape.config.dryRun` command-line flag is passed Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944	2020-12-07 13:18:16 +02:00
Aliaksandr Valialkin	9660774fd1	app/vmselect/graphite: remove duplicate `name` tag from `/tags/autoComplete/tags` handler Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942	2020-12-07 01:10:02 +02:00
Aliaksandr Valialkin	9d79d11a6c	lib/promscrape: mention in scrape error message that scrape errors can be disabled by `-promscrape.suppressScrapeErrors` command-line flag	2020-12-06 23:27:07 +02:00
Roman Khavronenko	22baf8fe25	dashboard: release to grafana.com (#941 )	2020-12-06 13:33:52 +02:00
Aliaksandr Valialkin	f6d32f99d7	lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set	2020-12-06 13:19:32 +02:00
Aliaksandr Valialkin	3d00613076	lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol	2020-12-06 12:00:28 +02:00
Roman Khavronenko	b53cf5d083	dashboard: Prometheus compatibility fix for `Storage full ETA` panel (#939 )	2020-12-06 01:19:20 +02:00
Aliaksandr Valialkin	80084d1827	docs/CHANGELOG.md: cut v1.49.0	2020-12-05 13:49:17 +02:00
Aliaksandr Valialkin	8dab061f51	vendor: `make vendor-update`	2020-12-05 12:45:41 +02:00
Aliaksandr Valialkin	f4f530d686	deployment/docker: update Go builder from v1.15.5 to v1.15.6 This fixes issues found in Go since v1.15.5 - see https://github.com/golang/go/issues?q=milestone%3AGo1.15.6+label%3ACherryPickApproved	2020-12-05 12:38:14 +02:00
Aliaksandr Valialkin	d242c2f2bd	app/vmselect/promql: add `count_eq_over_time(m[d], N)` and `count_ne_over_time(m[d], N)` for calculating the number of samples in `m` over `d` that are equal / not equal to `N`	2020-12-05 12:31:01 +02:00
Aliaksandr Valialkin	1430bbcf33	lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries Too low limit could result in unexpected errors when performing big number of blocking queries.	2020-12-05 12:15:47 +02:00
Aliaksandr Valialkin	528587deef	lib/flagutil: make golangci-lint happy by using strings.TrimPrefix instead of manual prefix removal via strings.HasPrefix	2020-12-03 22:07:26 +02:00
Aliaksandr Valialkin	bdac2171f1	all: do not print usage info for all the flags when incorrect command-line flag is passed This should improve usability for VictoriaMetrics apps that have big number of command-line flags, i.e. all the apps.	2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin	8cf76d8747	app/vmselect/promql: add `label_uppercase(q, label1, ... labelN)` and `label_lowercase(q, label1, ... labelN)` functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936	2020-12-03 21:46:18 +02:00
Aliaksandr Valialkin	732d354b90	vendor: `make vendor-update`	2020-12-03 20:21:54 +02:00
Aliaksandr Valialkin	96190f9d45	lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher	2020-12-03 20:14:48 +02:00
Aliaksandr Valialkin	4e4a93c586	lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header	2020-12-03 20:05:54 +02:00
Aliaksandr Valialkin	7a889f6850	lib/promscrape: code cleanup after `c6dee6c52d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-03 19:52:09 +02:00
Nikolay	0b302d33cb	Changes consul discovery api (#921 ) * adds consul watch api, it must reduce load on consul service with blocking wait requests, changed discoveryClient api with fetchResponseMeta callback. * small fix * fix after master merge * adds watch client at discovery utils * fixes consul watcher, changes namings, fixes data race * small typo fix * sanity fix * fix naming and service node update	2020-12-03 19:52:08 +02:00
Aliaksandr Valialkin	fca915dcf3	docs/Single-server-VictoriaMetrics.md: update `features` chapter according to the latest developments	2020-12-03 13:01:46 +02:00
Aliaksandr Valialkin	45c402ad8a	docs/Single-server-VictoriaMetrics.md: remove duplicate `provide` word	2020-12-03 09:48:54 +02:00
Aliaksandr Valialkin	11bbb3552d	app/vmselect/promql: `make fmt`	2020-12-02 21:34:15 +02:00
Aliaksandr Valialkin	dd96714a2c	docs/FAQ.md: add a link to https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f in `performance comparisons` section	2020-12-02 21:25:15 +02:00
Aliaksandr Valialkin	9e98a8f3d3	app/vmselect/promql: return `nan` from `minute(m)` when `m` equals to `nan` This aligns VictoriaMetrics behaviour with Prometheus behaviour. The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 20:16:40 +02:00
Aliaksandr Valialkin	def513355e	app/vmselect/promql: do not return `0` value from `sum_over_time(m[d])` when there are no samples on the given `d` window. This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.	2020-12-02 13:12:33 +02:00
Aliaksandr Valialkin	490c70a958	app/vmselect: return `metric` values from `time() cmp_op metric` query when `cmp_op` comparison is true This aligns MetricsQL behavior to Prometheus' one. The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 12:09:40 +02:00
Aliaksandr Valialkin	a8c5e2f0c5	vendor: upgrade github.com/VictoriaMetrics/metricsql from v0.7.2 to v0.7.3 This fixes parsing of hex numbers in MetricsQL such as 0x3b The bug has been detected at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/	2020-12-02 08:11:42 +02:00
Aliaksandr Valialkin	e6bc08436c	docs/Articles.md: add a link to https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870	2020-12-02 07:21:10 +02:00
Aliaksandr Valialkin	a34910e12c	docs/Articles.md: add a link to an article on how Percona PMM has been migrated from Prometheus to VictoriaMetrics	2020-12-02 07:21:08 +02:00
Aliaksandr Valialkin	4ef7158e89	app/vmselect/promql: return `nan` from `a >bool b` if `a` is `nan` in the same way as Prometheus does	2020-12-02 00:28:56 +02:00
Aliaksandr Valialkin	adf45b730c	app/vmselect/searchutils: return elapsed time in Deadline.String() output This should improve debuggability for error messages containing Deadline.String() output	2020-12-01 00:14:36 +02:00
Aliaksandr Valialkin	1dce37b2fa	app/vmbackup/snapshot: add missing status code check for the returned response when working with snapshot API Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929	2020-11-30 14:49:29 +02:00
Roman Khavronenko	d5ba66c303	Cluster dashboard (#931 ) * dashboard: add `Storage full ETA` panel Backport of https://github.com/VictoriaMetrics/VictoriaMetrics/pull/858 * dashboard: add `Storage reachability` panel	2020-11-30 11:30:11 +02:00
Aliaksandr Valialkin	8b5a38376d	app/vmbackup/snapshot: log url and response body on failed JSON response parsing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929	2020-11-29 12:16:08 +02:00
Nikolay	e4e33cb757	fixes checksum calculation (#928 ) * fixes checksum calculation, 'for' rule param wasnt marshal properly during checksum calculation * fixes error	2020-11-29 09:50:57 +02:00
Aliaksandr Valialkin	9eca96596f	lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test	2020-11-29 01:25:50 +02:00
Aliaksandr Valialkin	2385ac11c0	lib/promscrape: fix failing tests after `a906b3862f`	2020-11-29 01:25:49 +02:00
Aliaksandr Valialkin	2ed721e457	lib/protoparser/prometheus: properly parse OpenMetrics timestamps OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds. This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.	2020-11-27 14:54:36 +02:00
Aliaksandr Valialkin	2c0b1d5454	sync with README.md from single-node version	2020-11-27 13:23:16 +02:00
Aliaksandr Valialkin	6bd9fe4e77	docs/Articles.md: add https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f	2020-11-27 10:25:00 +02:00
Aliaksandr Valialkin	3bb9bf33d6	lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets	2020-11-26 18:32:16 +02:00
Aliaksandr Valialkin	af667c59c1	all: typo fix: thouthand->thousand	2020-11-26 13:34:05 +02:00
Aliaksandr Valialkin	3de102bcf1	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.8 to v1.0.9	2020-11-26 13:27:23 +02:00
Aliaksandr Valialkin	3dd2282ed9	lib/promscrape: release http response non-200 status code	2020-11-26 13:25:25 +02:00
Aliaksandr Valialkin	e6447e7588	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.7 to v1.0.8	2020-11-26 12:13:09 +02:00
Aliaksandr Valialkin	81fadba0b2	docs/CHANGELOG.md: cut v1.48.0 release	2020-11-26 02:06:11 +02:00
Aliaksandr Valialkin	b542df9ab5	docs/CHANGELOG.md: add a link to Netflix Eureka - https://github.com/Netflix/eureka	2020-11-26 01:36:30 +02:00
Aliaksandr Valialkin	3f52e59efe	app/{vmagent,victoria-metrics}: add `-dryRun` option and make more clear handling for `-promscrape.config.dryRun`	2020-11-25 23:01:39 +02:00
Aliaksandr Valialkin	ed06990609	app/vmagent: do not enable -promscrape.config.strictParse when -dryRun command-line flag is set Users can specify -promscrape.config.strictParse if -promscrape.config shouldn't contain unknown config entries	2020-11-25 22:27:41 +02:00
Aliaksandr Valialkin	8f8727cb65	lib/mergeset: tune the number of rawItemsBlocks to merge at once 512 blocks give higher ingestion performance and slightly lower memory usage	2020-11-25 21:53:15 +02:00
Aliaksandr Valialkin	8fcd87a6a5	lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset	2020-11-25 21:20:02 +02:00
Aliaksandr Valialkin	03002f1fe1	lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention This should simplify debugging when the source of the metric with unexpected timestamp must be found.	2020-11-25 14:44:29 +02:00
Aliaksandr Valialkin	4848a05924	lib/storage: typo fix in error message: allowd->allowed	2020-11-25 14:15:54 +02:00
Aliaksandr Valialkin	26e699c440	lib/protoparser/prometheus: properly parse "infinity" values in OpenMetrics format Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924	2020-11-24 19:02:50 +02:00
Aliaksandr Valialkin	a8b5a6e517	lib/logger: disable rate limiting for error and warn logs by default	2020-11-24 12:42:07 +02:00
Aliaksandr Valialkin	7f3e884a31	all: spelling fix: superflouos->superfluous. This is a follow-up for `0acdab3ab9`	2020-11-24 12:42:04 +02:00
BigFish	3159b41689	Update main.go (#922 ) fix spelling mistake	2020-11-24 12:36:47 +02:00
Aliaksandr Valialkin	284d805895	docs/CHANGELOG.md: mention that `/tags/delSeries` handler is supported after `f0c207fae2`	2020-11-24 12:30:36 +02:00
Aliaksandr Valialkin	dad8b76a0e	lib/protoparser/prometheus: properly parse metrics with exemplars Examplars have been introduced in OpenMetrics - see https://github.com/OpenObservability/OpenMetrics/blob/master/OpenMetrics.md#exemplars-1 Previously VictoriaMetrics couldn't parse the following metric foo{bar="baz"} 123 # exemplar here This commit fixes this. Note that VictoriaMetrics ignores the exemplar as for now.	2020-11-24 12:30:34 +02:00
Aliaksandr Valialkin	768fd8c3d9	docs/Articles.md: add recent articles about VictoriaMetrics	2020-11-24 12:30:30 +02:00
Aliaksandr Valialkin	2cc288c023	app/vmbackup: cosmetic fixes	2020-11-23 17:10:13 +02:00
Aliaksandr Valialkin	8b82f9d8b8	lib/promscrape: expose `__meta_ec2_ipv6_addresses` label for `ec2_sd_config` like Prometheus will do in the next release	2020-11-23 16:57:03 +02:00
Aliaksandr Valialkin	c2186279b7	lib/promscrape: add `filters` option to `dockerswarm_sd_config` like Prometheus did in v2.23.0	2020-11-23 16:27:33 +02:00
Aliaksandr Valialkin	e1297c0b78	app/vmselect: add `/tags/delSeries` handler from Graphite Tags API See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb	2020-11-23 15:32:14 +02:00
Aliaksandr Valialkin	3d2ce31cad	app/vmselect/netstorage: code readability improvement: rename RequestErrors to Errors	2020-11-23 15:00:15 +02:00
Aliaksandr Valialkin	433ae806ac	app/vmselect: implement `/tags/tagSeries` and /tags/tagMultiSeries` in order to be consistent with single-node VictoriaMetrics	2020-11-23 14:57:08 +02:00
Aliaksandr Valialkin	7987129baa	app/vmselect/netstorage: move common code for requests execution on all the storage nodes to startStorageNodesRequest func	2020-11-23 10:51:48 +02:00
Aliaksandr Valialkin	25a57ced6c	app/vmselect/netstorage: prevent from data races in ProcessSearchQuery and in Export funcs when -replicationFactor > 1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-11-23 10:25:51 +02:00
Aliaksandr Valialkin	f4fd917e4f	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin	1dcb438c3b	app/vmselect/netstorage: typo fix after `990eb29a9b`	2020-11-23 01:09:43 +02:00
Aliaksandr Valialkin	85eecf5801	app/vmselect/netstorage: add `-replicationFactor` command-line flag for reducing query duration when a part of vmstorage nodes are temporarily slow and/or temporarily unavailable Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-11-23 00:39:53 +02:00
Aliaksandr Valialkin	990eb29a9b	app/vmselect/netstorage: move common code for collecting query results from vmstorage nodes to collectResults function	2020-11-23 00:16:02 +02:00
Aliaksandr Valialkin	d892d63204	lib/promscrape: hint that -enableTCP6 command-line flag can be used for connecting to IPv6 addresses	2020-11-21 14:39:05 +02:00
Aliaksandr Valialkin	8608e956dd	lib/promscrape/discovery/eureka: follow-up after `eec76718e9`	2020-11-20 14:02:14 +02:00
Nikolay	bb2bcb9725	Adds eureka service discovery (#913 ) * Adds eureka service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851 Netflix service discovery for AWS * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-20 14:02:13 +02:00
John Belmonte	7ac49ac176	MetricsQL docs: parameter consistency (#915 ) * MetricsQL docs: parameter consistency if I understand correctly: * `fun(q)` - fun takes instant vector * `fun(m[d])` - fun takes range vector * Update docs/MetricsQL.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-20 11:43:07 +02:00
Aliaksandr Valialkin	59aa57efbc	vendor: `make vendor-update`	2020-11-19 19:21:23 +02:00
Aliaksandr Valialkin	5e39bedf40	docs/CHANGELOG.md: mention that slow query log now contains remote client address	2020-11-19 12:41:26 +02:00
Aliaksandr Valialkin	8e1f657ef9	lib/logger: follow-up for `09105ff49c`	2020-11-19 12:37:05 +02:00
Nikolay	f54a5f3868	Adds log suppression per caller (#908 ) * Adds log suppression per caller https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905 * fixes style and report message	2020-11-19 12:19:20 +02:00
Aliaksandr Valialkin	e72ccc9239	app/vmselect: add remoteAddr to slow query log in order to improve debuggability This will simplify identifying the client that sends slow queries to VictoriaMetrics.	2020-11-18 20:40:02 +02:00
Aliaksandr Valialkin	2af08d6e97	docs/CHANGELOG.md: mention about `snap install victoriametrics`	2020-11-18 19:50:38 +02:00
Aliaksandr Valialkin	5331349efb	docs: make `snap install victoriametrics` more prominent in docs	2020-11-18 19:45:10 +02:00
Aliaksandr Valialkin	f372148203	docs/Single-server-VictoriaMetrics.md: an attempt to fix markdown formatting in `Graphite Tags API` section	2020-11-18 14:41:48 +02:00
Aliaksandr Valialkin	e7976363f9	docs: lowercase `adidas` trademark according to their request	2020-11-18 13:47:56 +02:00
Aliaksandr Valialkin	a5413aa438	docs/Cluster-VictoriaMetrics.md: adjust RAM sizing recommendations for vmstorage nodes It is recommended to have at least of 50% of free RAM on vmstorage nodes in order handle possible RAM usage spikes during rolling upgrade for vmstorage nodes when time series are re-routed from temporarily unavailable node to the remaining active nodes.	2020-11-18 13:02:38 +02:00
Aliaksandr Valialkin	a0a34e2a26	docs/Single-server-VictoriaMetrics.md: make consistent section title sizes	2020-11-18 12:36:22 +02:00
Aliaksandr Valialkin	0895b7f411	lib/logger: add `-loggerWarnsPerSecondLimit` command-line flag for rate limiting of WARN log messages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905	2020-11-18 03:43:17 +02:00
Aliaksandr Valialkin	bf9b6b77c8	docs/Single-server-VictoriaMetrics.md: mention that `/internal/force_flush` endpoint is mostly needed for testing and debugging	2020-11-18 01:37:39 +02:00
Aliaksandr Valialkin	ea4afb201b	app/vmselect/netstorage: typo fix in a comment inside SearchMetricNames func	2020-11-18 01:35:37 +02:00
Aliaksandr Valialkin	c6adcafedb	app/vminsert: export `vm_rpc_vmstorage_is_reachable` metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes	2020-11-17 22:13:26 +02:00
Aliaksandr Valialkin	a00df790b1	docs/CHANGELOG.md: cut v1.47.0 release	2020-11-16 21:00:21 +02:00
Aliaksandr Valialkin	d29997dab6	vendor: `make vendor-update`	2020-11-16 20:52:18 +02:00
Aliaksandr Valialkin	e8ef94db4b	Makefile: add `-d` flag to `go get` in `vendor-update` target This should skip unnecessary build step for the updated packages	2020-11-16 20:52:15 +02:00
Aliaksandr Valialkin	5454137504	docs/Cluster-VictoriaMetrics.md: `make docs-sync` after `57dc152e9d`	2020-11-16 20:20:19 +02:00
Nikita Babushkin	57dc152e9d	fix typo in readme.md (#900 )	2020-11-16 20:18:25 +02:00
Aliaksandr Valialkin	7d76fdedcc	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin	a9287cf564	lib/storage: do not pass (accountID, projectID) to SearchTagNames(), since they are already passed via tfss	2020-11-16 18:04:30 +02:00
Aliaksandr Valialkin	911c6d3bcd	app/vmselect: add missing graphite prefix to /tags/autoComplete/{tags,values}	2020-11-16 18:04:24 +02:00
Aliaksandr Valialkin	f7f866d83b	app/vmselect/netstorage: typo fix	2020-11-16 15:54:45 +02:00
Aliaksandr Valialkin	59fb75717e	app/vmselect/netstorage: apply Graphite filter after substituting `__name__` with `name`	2020-11-16 15:50:53 +02:00
Aliaksandr Valialkin	6bcbdb18fb	docs/CHANGELOG.md: mention about Graphite Tags API implementation	2020-11-16 15:34:56 +02:00
Aliaksandr Valialkin	eb763bcb9d	app/vmselect/graphite: add `/tags/autoComplete/values` handler from Graphite Tags API	2020-11-16 15:29:29 +02:00
Aliaksandr Valialkin	f2f16d8e79	app/vmselect/graphite: add `/tags/autoComplete/tags` handler from Graphite Tags API See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support	2020-11-16 14:58:10 +02:00
Aliaksandr Valialkin	2f4421b86c	app/vmselect/prometheus: return `__name__` label if `match[]` query to `/api/v1/labels` matches at least a single time series	2020-11-16 13:54:50 +02:00
Aliaksandr Valialkin	852aed62f7	app/vmselect/prometheus: improve performance for `/api/v1/labels` and `/api/v1/label/<labelName>/values` on time ranges exceeding one day when `match[]` query arg is set	2020-11-16 13:46:51 +02:00
Aliaksandr Valialkin	e969346e3e	app/vmselect/prometheus: fix deadlock in `/api/v1/series` on a time range exceeding one day	2020-11-16 13:30:57 +02:00
Aliaksandr Valialkin	ac7460abdd	lib/storage: add a test for Storage.SearchMetricNames	2020-11-16 13:18:48 +02:00
Aliaksandr Valialkin	467ed68a37	docs/CHANGELOG.md: mention about improved performance for `/api/v1/series` on a time range exceeding one day	2020-11-16 12:52:33 +02:00
Aliaksandr Valialkin	eea1be0d5c	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:52:23 +02:00
Aliaksandr Valialkin	97100b1d42	app/vmselect/graphite: apply filter then limit	2020-11-16 12:52:18 +02:00
Aliaksandr Valialkin	5889273920	app/vmselect/graphite: add `/tags/<tag_name>` handler for Graphite Tags API	2020-11-16 03:41:41 +02:00
Aliaksandr Valialkin	99cb1a70cf	app/vmselect/graphite: add `/tags` handler from Graphite Tags API See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 02:57:20 +02:00
Aliaksandr Valialkin	4be5b5733a	app/vminsert: add `/tags/tagSeries` and `/tags/tagMultiSeries` handlers from Graphite Tags API See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb	2020-11-16 02:40:04 +02:00
Aliaksandr Valialkin	9ec964bff8	lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page	2020-11-16 00:44:54 +02:00
Aliaksandr Valialkin	2ac5f00d98	app/vmselect: propagate errors from `vmstorage` to response to the client if `-search.denyPartialResponse` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891 This commit also adds `"isPartial":{true\|false}` field to `/api/v1/*` responses. `"isPartial":true` is set when the response is based on a partial data because some of vmstorage nodes weren't available during query processing.	2020-11-14 13:20:10 +02:00
Aliaksandr Valialkin	882e2e2099	app/vminsert/netstorage: return 503 status code to client when all the vmstorage nodes are unavailable Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-14 00:44:41 +02:00
Aliaksandr Valialkin	c329aacedf	docs/Single-server-VictoriaMetrics.md: document `/internal/force_flush` endpoint	2020-11-13 18:43:40 +02:00
Aliaksandr Valialkin	f19547039a	docs/Single-server-VictoriaMetrics.md: explain why recently inserted data may be unavailable for querying for a few seconds	2020-11-13 18:34:31 +02:00
Aliaksandr Valialkin	f80d6473e1	lib/protoparser/promremotewrite: log the time spent on unsuccessful data read from the network This should help with debugging `connection timed out` errors.	2020-11-13 17:49:21 +02:00
Aliaksandr Valialkin	1364dfdd8c	docs/CHANGELOG.md: mentioned that Go builder has been updated from v1.15.4 to v1.15.5 See `3fa9ab4a49` for details.	2020-11-13 16:22:18 +02:00
Vasily	8ba168f3be	Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig (#796 ) * Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig * Add omitempty annotation to all the default/optional values * Fix annotations after review	2020-11-13 16:17:03 +02:00
Aliaksandr Valialkin	cc4da051f3	docs/Single-server-VictoriaMetrics.md: sync with single-node README.md	2020-11-13 16:03:50 +02:00
Aliaksandr Valialkin	8f42e59e05	app/vmselect/promql: remove spikes from `increase()` and `delta()` results on time series with spare irregular data points Do not take into account spare data point value if the next point will is located too far from the current point. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 15:23:37 +02:00
Aliaksandr Valialkin	da6d82a8dd	app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() This should remove unexpected spikes at the end of gaps. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894	2020-11-13 14:59:32 +02:00
Aliaksandr Valialkin	d21b1606a1	lib/protoparser/opentsdbhttp: increment errors counter on unmarshal errors This is a follow-up for `149c0c4a6d`	2020-11-13 13:23:11 +02:00
Aliaksandr Valialkin	4d9501a0c4	vendor: `make vendor-update`	2020-11-13 13:08:41 +02:00
Aliaksandr Valialkin	22c1e29284	lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 13:05:34 +02:00
Aliaksandr Valialkin	9dfe00c962	lib/protoparser/promremotewrite: synchronously process Prometheus remote_write requests There is no reason in processing these requests asynchronously in the face of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896 Synchronous processing code is easier to read and understand than the previous async code	2020-11-13 12:17:32 +02:00
Aliaksandr Valialkin	739b88c1e4	lib/protoparser/promremotewrite: forward errors, which can occur during data ingestion, to the caller of ParseStream, so it could properly return HTTP 503 status code on non-nil error Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 11:00:41 +02:00
Aliaksandr Valialkin	13b547f218	deployment/docker: update Go builder from v1.15.4 to v1.15.5 This fixes the following possible issues in Go - https://github.com/golang/go/issues?q=milestone%3AGo1.15.5+label%3ACherryPickApproved	2020-11-13 11:00:40 +02:00
Aliaksandr Valialkin	7ceaf4ba8f	all: consistently return text-based HTTP responses with charset=utf-8 This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:30:21 +02:00
faceair	64e99744f1	add charset on targets response (#897 )	2020-11-13 10:18:13 +02:00
Aliaksandr Valialkin	f7a6ae3d11	docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into `Relabeling` section	2020-11-12 12:27:13 +02:00
Aliaksandr Valialkin	069979c367	docs/vmagent.md: typo fix	2020-11-11 16:05:04 +02:00
Aliaksandr Valialkin	7a0094adae	docs/vmagent.md: add `Configuration update` section	2020-11-11 16:01:21 +02:00
Aliaksandr Valialkin	94842ed942	docs/Single-server-VictoriaMetrics.md: document `-search.treatDotsAsIsInRegexps` command-line option	2020-11-11 14:59:38 +02:00
immerrr again	1ec1a9f27f	app/vmstorage: add "/internal/force_flush" endpoint (#893 )	2020-11-11 14:46:37 +02:00
Aliaksandr Valialkin	6b979ea5a7	docs/Single-server-VictoriaMetrics.md: small clarifications in VictoriaMetrics features	2020-11-11 13:48:16 +02:00
Aliaksandr Valialkin	20d0ead2f2	docs/Single-server-VictoriaMetrics.md: update the link to enterprise features	2020-11-11 13:42:42 +02:00
Aliaksandr Valialkin	779f591bcc	docs/Single-server-VictoriaMetrics.md: mention that `/api/v1/status/tsdb` handler accepts `topN` and `date` query args	2020-11-11 13:38:26 +02:00
Aliaksandr Valialkin	8033e4885b	docs/Cluster-VictoriaMetrics.md: mention about optional topN and date query args for `/api/v1/status/tsdb` handler	2020-11-11 13:35:13 +02:00
Aliaksandr Valialkin	4f2c5877db	app/vmselect: add `-search.treatDotsAsIsInRegexps` command-line flag for automatic escaping of dots in regexp label filters	2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin	0769f86a7e	docs/Single-server-VictoriaMetrics.md: clarify which directories can be removed when recovering from data corruption	2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin	346181fd48	docs/Single-server-VictoriaMetrics.md: add a hint that case studies can be read by clicking on the corresponding link	2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin	a78bf34ff3	app/vmselect: do not return `isPartialResponse=true` when all the storageNodes return errors	2020-11-10 18:48:57 +02:00
Aliaksandr Valialkin	697fd44158	lib/promscrape: make a copy of ScrapeWork from discovered []ScrapeWork slice instead of referring to an item in this slice This should prevent from holding previously discovered []ScrapeWork slices when a part of discovered targets changes over time. This should reduce memory usage for the case when big number of discovered scrape targets changes over time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 16:13:31 +02:00
Aliaksandr Valialkin	2ec02b7bdb	lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets This should reduce load on GC a bit when discovering big number of scrape targets	2020-11-10 15:57:43 +02:00
Aliaksandr Valialkin	8f3339fa81	app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin	6385432611	app/vmselect: typo fix in a description for `-search.minStalenessInterval`: mimimum->minimum	2020-11-10 01:18:59 +02:00
Aliaksandr Valialkin	6ae9d79f6b	docs/CHANGELOG.md: mention abot explicit setting of extra labels in alert entities (see `3adf8c5a6f`) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-10 00:41:31 +02:00
Aliaksandr Valialkin	ad82b6ead8	Move CHANGELOG.md to docs/CHANGELOG.md	2020-11-10 00:37:15 +02:00
Roman Khavronenko	4fd2b6cd16	vmalert: explicitly set extra labels to alert entities (#886 ) The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, time series always contained only original labels and alert ID was generated from sorted labels key-values. Extra labels didn't affect the generated ID and were applied on the following actions: - templating for Summary and Annotations; - persisting state via remote write; - restoring state via remote read. Such behaviour caused difficulties on restore procedure because extra labels had to be dropped before checking the alert ID, but that not always worked. Consider the case when expression returns the following time series `up{job="foo"}` and rule has extra label `job=bar`. This would mean that restored alert ID will be always different to the real time series because of collision. To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't store original labels anymore. However, this could result into a new error situation. Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`, while rule has extra label `job=bar`. In such case, applying extra labels will result into two identical time series and `vmalert` will return error: `result contains metrics with the same labelset after applying rule labels` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-10 00:27:56 +02:00
Aliaksandr Valialkin	a8562d643b	lib/promscrape: add `-promscrape.dropOriginalLabels` command-line flag for reducing memory usage when discovering big number of scrape targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin	aa3e46a02d	lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-09 10:55:03 +02:00
Aliaksandr Valialkin	96e9deecbc	docs/Single-server-VictoriaMetrics.md: typo fix	2020-11-08 13:40:52 +02:00
Aliaksandr Valialkin	a8ad3091e3	CHANGELOG.md: cut v1.46.0	2020-11-07 17:54:10 +02:00
Aliaksandr Valialkin	d2296b7e09	docs/CHANGELOG.md: `make docs-sync`	2020-11-07 16:30:41 +02:00
Aliaksandr Valialkin	b8083b7659	lib/promscrape: clean references to label name and label value strings after applying per-target relabeling This should reduce memory usage when per-target relabeling creates big number of temporary labels with long names and/or values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin	b4efe626d7	lib/promscrape/discovery/kubernetes: `go fmt`	2020-11-07 13:04:09 +02:00
Aliaksandr Valialkin	92bc1afcee	lib/promscrape/discovery/kubernetes: reduce memory usage for labels when discovering big number of scrape targets by using string concatenation instead of fmt.Sprintf Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 13:03:01 +02:00
Aliaksandr Valialkin	535fea3d11	lib/promscrape: eliminate data race in `stream parse` mode Previously `-promscrape.streamParse` mode could result in garbage labels for the scraped metrics because of data race. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247	2020-11-07 12:45:52 +02:00
Aliaksandr Valialkin	6ca5a94359	docs/CHANGELOG.md: add `CHANGELOG` header	2020-11-07 01:14:49 +02:00
Aliaksandr Valialkin	4b57604657	deployment/docker: update Go builder from v1.15.3 to v1.15.4 This fixes issues found in Go - see https://github.com/golang/go/issues?q=milestone%3AGo1.15.4+label%3ACherryPickApproved	2020-11-07 00:49:34 +02:00
Aliaksandr Valialkin	45958d16f6	vendor: `make vendor-update`	2020-11-06 13:57:06 +02:00
Aliaksandr Valialkin	6866e4ab25	deployment/docker: update root image from alpine:3.12 to alpine:3.12.1	2020-11-06 13:54:30 +02:00
Aliaksandr Valialkin	efebc3b6fb	app/vmselect/promql: code cleanup after `43823addea`	2020-11-06 01:31:33 +02:00
n4mine	3127aa92b5	app/vmselect/promql: fix when the parameter of `maxValue()`, `minValue()` leading by `NaN`. it will cause `{top,bottom}k_{max,min}` return inappropriate result (#883 )	2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin	8accbc14d8	docs/Articles.md: add a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2	2020-11-05 19:18:13 +02:00
Aliaksandr Valialkin	6ce80425a6	docs/MetricsQL.md: fix a link to `with templates` page	2020-11-05 16:12:59 +02:00
Aliaksandr Valialkin	60747c5f14	docs/Articles.md: move third-party articles to the top	2020-11-05 15:03:50 +02:00
Aliaksandr Valialkin	bf28bc3792	docs: add CHANGELOG.md	2020-11-05 14:59:17 +02:00
Aliaksandr Valialkin	53193ca2bc	CHANGELOG.md: add a link to issue related to `/ready` endpoint Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/875	2020-11-05 14:56:59 +02:00
Aliaksandr Valialkin	8a577568e3	CHANGELOG.md: mention about `/ready` endpoint in `vmagent`	2020-11-05 14:54:38 +02:00
Aliaksandr Valialkin	767231f41f	app/vmstorage/transport: properly handle request to labelValuesOnTimeRange	2020-11-05 02:08:04 +02:00
Aliaksandr Valialkin	72011bcc45	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin	f2bff64933	lib/storage: remove data race when updating rowsDeleted	2020-11-05 01:19:30 +02:00
Aliaksandr Valialkin	c5e6c5f5a6	app/vmselect: optimize querying for `/api/v1/labels` and `/api/v1/label/<name>/values` when `start` and `end` args are set	2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin	1336e47c86	docs/vmagent.md: update after `4c808d58bf`	2020-11-04 20:33:49 +02:00
Nikolay	5b235b902b	Adds ready probe (#874 ) * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:33:48 +02:00
Aliaksandr Valialkin	d8a7186019	docs/CaseStudies.md: add case study for Idealo.de	2020-11-04 20:19:26 +02:00
Aliaksandr Valialkin	2cd86d0220	lib/promscrape: docs update after `e4182dd896` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 17:13:34 +02:00
Nikolay	d0a9b24c5a	reduces memory usage for vmagent, (#880 ) * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin	692f5d7bca	app/vmselect: reduce memory usage when query touches big number of time series	2020-11-04 16:48:10 +02:00
Aliaksandr Valialkin	c736339843	lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively Previously such blocks were cleaned after they weren't accessed during 10 minutes. Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.	2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin	f35aafb6a5	Revert "lib/promscrape: add `-promscrape.dropOriginalLabels` command-line flag for reducing memory usage when discovering big number of scrape targets" This reverts commit `b08c6f5144`.	2020-11-04 11:45:38 +02:00
Aliaksandr Valialkin	407a46c11e	Revert "docs/vmagent.md: mention about `-promscrape.dropOriginalLabels`" This reverts commit `1a80acc712`.	2020-11-04 11:45:35 +02:00
Aliaksandr Valialkin	1a80acc712	docs/vmagent.md: mention about `-promscrape.dropOriginalLabels`	2020-11-04 11:16:16 +02:00
Aliaksandr Valialkin	b08c6f5144	lib/promscrape: add `-promscrape.dropOriginalLabels` command-line flag for reducing memory usage when discovering big number of scrape targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 11:09:05 +02:00
Aliaksandr Valialkin	c046735571	lib/promscrape: reduce memory allocations in promLabelsString() function This should help with reducing memory usage in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 10:38:59 +02:00
Aliaksandr Valialkin	c0bd208c77	lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification	2020-11-03 15:32:09 +02:00
Aliaksandr Valialkin	887a3c317f	app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop	2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin	1b9778a756	lib/storage: remove unneeded fmt.Sprintf	2020-11-03 14:21:04 +02:00
John Belmonte	8653e2658e	add short_version label to vm_app_version metric (#877 ) * add short_version label to vm_app_version metric use case: Version panel of Grafana dashboard should use a live query, but currently it uses a template query which becomes stale. Grafana is not able to preform regex substitution on labels. * Update metrics.go * fix compile	2020-11-03 14:12:42 +02:00
Aliaksandr Valialkin	40a4f5ded4	docs/MetricsQL.md: add missing whitespace	2020-11-02 23:49:45 +02:00
Aliaksandr Valialkin	dbd1789479	vendor: make vendor-update	2020-11-02 22:04:35 +02:00
Aliaksandr Valialkin	f3a7e6f6e3	lib/storage: remove obsolete code	2020-11-02 19:17:30 +02:00
Aliaksandr Valialkin	9715d9a3a8	CHANGELOG.md: cut v1.45.0	2020-11-02 04:00:41 +02:00
Aliaksandr Valialkin	9df946517c	vendor: `make vendor-update`	2020-11-02 02:40:17 +02:00
Aliaksandr Valialkin	c76242d52d	CHANGELOG.md: add a link to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-02 02:29:11 +02:00
Aliaksandr Valialkin	66de02fbb4	app/vmselect/promql: allow dropping trailing sample only for `default_rollup` function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850	2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin	3ed9f1d5a9	lib/promscrape: properly handle response body after 301 redirect Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:59 +02:00
Aliaksandr Valialkin	bc7db0bf4f	CHANGELOG.md: mention about packets drop in vmagent like Prometheus does	2020-11-02 00:47:05 +02:00
Aliaksandr Valialkin	ca2e0f1e04	app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code This makes consistent the behaviour with Prometheus. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873	2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin	6b623eba02	app/vmselect/promql: go fmt	2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin	7c0b658865	app/vmselect/promql: do not drop trailing datapoints for instant queries Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-11-02 00:12:53 +02:00
Aliaksandr Valialkin	ebe2782dcf	CHANGELOG.md: mention about recently added bugfixes	2020-11-01 23:33:50 +02:00
Roman Khavronenko	333675875f	vmalert: skip automatically added labels on alerts restore (#871 ) Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correctly purged on restore event and affected alert's ID uniqueness. This commit removes `alertgroup` label in restore function. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-01 23:26:00 +02:00
Nikolay	e8fe618bbb	fixes panic at scrape error body formating, (#868 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-11-01 23:25:23 +02:00
Nikolay	058f49de57	adds leading forward slash check for scrapeURL path (#855 ) * fixes in-consistency with prometheus behaviour for scrape targets url path. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835	2020-11-01 23:22:16 +02:00
kreedom	40172c0721	vmbackup fix panic when no origin fs given (#859 ) * use fsnil when no origin fs	2020-11-01 23:17:01 +02:00
Aliaksandr Valialkin	ed724d25ba	lib/promscrape: add `stream parse` mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin	901514be88	lib/storage: drop more samples outside the given retention during background merge Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-31 20:44:47 +02:00
Aliaksandr Valialkin	abdf22e0bb	app/vmagent: expose `/api/v1/targets` page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin	c4464594b7	app/vmselect/promql: allow passing optional third argument to `topk_` and `bottomk_` functions in order to obtain sum of time series outside top/bottom K	2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin	7599e5c835	lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*	2020-10-20 20:09:52 +03:00
Aliaksandr Valialkin	9c5cd5a6c5	lib/storage: code cleanup after `5bfd4e6218`	2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin	0db7c2b500	app/vmstorage: support for `-retentionPeriod` smaller than one month Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:42:46 +03:00
Aliaksandr Valialkin	b915bf01e4	CHANGELOG.md: mention about recently added changes	2020-10-20 14:42:43 +03:00
Aliaksandr Valialkin	5518b11720	lib/memory: do not print trailing zeroes in logs for `-memory.allowedPercent` command-line flag	2020-10-20 14:42:37 +03:00
Aliaksandr Valialkin	abdf020e22	docs/vmrestore.md: `make docs-sync`	2020-10-20 10:49:22 +03:00
kreedom	4526cf92d3	vmalert - add dryRun (#842 ) vmalert: add `dryRun` flag for rules validation without running the service	2020-10-20 10:49:22 +03:00
Seva Poliakov	e6bf9eaac7	Fix typo in vnrestore readme	2020-10-20 10:49:22 +03:00
faceair	0093ee3cd9	disable response compression on websocket (#841 )	2020-10-17 13:33:37 +03:00
Aliaksandr Valialkin	9b91305a31	docs/MetricsQL.md: small clarifications	2020-10-17 12:01:33 +03:00
Aliaksandr Valialkin	ee2902ddaf	app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series Now trailing data points are additionally dropped for time series with a single raw sample Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-17 10:44:26 +03:00
Aliaksandr Valialkin	efb1989193	lib/storage: small code adjustements after `d2960a20e0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-10-17 01:17:12 +03:00
faceair	8ddf089deb	evaluate the execution cost of all tag filters (#824 ) * evaluate the execution cost of all tag filters * fix suffixes typo	2020-10-17 01:13:20 +03:00
Aliaksandr Valialkin	35791d9b29	CHANGELOG.md: mention about improved openstack endpoint handling Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-16 23:06:23 +03:00
Nikolay Khramchikhin	a5d842caf8	fixes openstack api endpoint with suffix trim adds openstack (#840 ) api v2.0 check Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-16 23:01:45 +03:00
Aliaksandr Valialkin	8d87b57fbf	deployment/docker: update Go builder from Go1.15.2 to Go1.15.3 This should fix potential issues related to Go runtime - see https://github.com/golang/go/issues?q=milestone%3AGo1.15.3+label%3ACherryPickApproved	2020-10-16 15:09:30 +03:00
Aliaksandr Valialkin	cde26141c0	vendor: `make vendor-update`	2020-10-16 15:05:54 +03:00
Aliaksandr Valialkin	da48a5a65c	CHANGELOG.md: describe added optimization cases from `96cdfcba50`	2020-10-16 12:59:48 +03:00
Aliaksandr Valialkin	8c027ba8a4	vendor: update github.com/VictoriaMetrics/metricsql from v0.7.1 to v0.7.2 The new release of github.com/VictoriaMetrics/metricsql adds more optimizations for `foo{filters1} op bar{filters2}`: * rollup_func(foo[d]) op bar{filters} * transform_func(foo) op bar{filters} * num_or_scalar op bar op baz{filters}	2020-10-16 12:55:27 +03:00
Aliaksandr Valialkin	6227b71a67	docs: add a link to https://smarketshq.com/monitoring-kubernetes-clusters-41a4b24c19e3 article about VictoriaMetrics	2020-10-16 09:07:55 +03:00
Aliaksandr Valialkin	bc446ec62a	docs/Single-server-VictoriaMetrics.md: update docs	2020-10-14 13:27:01 +03:00
Aliaksandr Valialkin	1634415441	docs/CaseStudies.md: actualize numbers for Wix.com	2020-10-14 13:07:41 +03:00
Aliaksandr Valialkin	e9fec0e5b8	docs/vmalert.md: `make docs-sync`	2020-10-13 18:35:00 +03:00
Roman Khavronenko	d6155a3f33	vmalert: update docs to highlight the state restore requirements; (#833 ) Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830	2020-10-13 18:34:00 +03:00
Aliaksandr Valialkin	4727bad124	CHANGELOG.md: cut v1.44.0 release	2020-10-13 16:59:47 +03:00
Aliaksandr Valialkin	de9d3f1332	docs/Cluster-VictoriaMetrics.md: clarify RAM requirements for `vmstorage` nodes	2020-10-13 16:47:25 +03:00
Aliaksandr Valialkin	b9a4601c97	app/vmselect/promql: return a single time series at max from `absent()` function like Prometheus does	2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin	217c192c88	app/vmselect/promql: improve time series staleness detection This should prevent from double counting for time series at the time when it changes label. The most common case is in K8S, which changes pod uid label with each new deployment. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin	f877e703c8	app/vmselect/promql: fix `mode_over_time` calculations Previously `mode_over_time` could return garbage due to improper shuffling of input data points.	2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin	d884ab13dc	app/vmselect/prometheus: fix golangci-lint warning	2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin	0867dea5fc	app/vmselect: add ability to export data in CSV format via `/api/v1/export/csv`	2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin	6105756b26	CHANGELOG.md: mention about added Docker Swarm service discovery	2020-10-12 16:17:42 +03:00
Aliaksandr Valialkin	938b3b7ed1	lib/promscrape: code prettifying after `9bd9f67718`	2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin	7f96712b38	Adds dockerswarm sd (#818 ) * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin	ac525462ce	docs/MetricsQL.md: mention that VictoriaMetrics keeps metric names after applying functions which dont change time series meaning	2020-10-12 13:25:28 +03:00
Aliaksandr Valialkin	ba4bfc2bdf	CHANGELOG.md: mention that VictoriaMetrics keeps metric names when applying functions which don't change time series meaning	2020-10-12 13:02:47 +03:00
Aliaksandr Valialkin	2d03d0e2dd	app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning Functions are: * keep_last_value * keep_next_value * interpolate * running_min * running_max * running_avg * range_min * range_max * range_avg * range_first * range_last * range_quantile * smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	3881c84afe	Revert "app/vmselect/promql: remove metric name after applying `ceil`, `floor` and `round` functions in order to be more consistent with Prometheus" This reverts commit `ac45082216`. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric names: * ceil * floor * round Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	79d70480b7	Revert "app/vmselect/promql: remove metric name after applying `clamp_min` and `clamp_max` functions in order to be consistent with Prometheus" This reverts commit `bb61a4769b`. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * clamp_min * clamp_max Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	8c37b63ea9	Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" This reverts commit `e5202a4eae`. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * max_over_time * min_over_time * avg_over_time * quantile_over_time * geomean_over_time * mode_over_time * holt_winters * predict_linear Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	85d507f71d	docs/Single-server-VictoriaMetrics.md: add missing whitespace	2020-10-09 20:56:45 +03:00
Aliaksandr Valialkin	de1c07b937	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin	bf6d523bef	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin	00b5568ca4	CHANGELOG.md: update with recent changes	2020-10-09 14:22:17 +03:00
Aliaksandr Valialkin	9b7ce5d004	app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in `vm_rows_inserted_total` and `vmagent_rows_inserted_total` metrics	2020-10-09 13:38:49 +03:00
Aliaksandr Valialkin	d2e917d1cb	app/vmstorage: add `vm_rows_added_to_storage_total` metric, which shows the total number of rows added to storage since app start	2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin	4b1c401790	app/vmalert: accept days, weeks and years in `for:` part of config like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817	2020-10-08 20:13:20 +03:00
Aliaksandr Valialkin	a93c62cd60	lib/promscrape: fix tests after `71ea4935de`	2020-10-08 19:32:48 +03:00
Aliaksandr Valialkin	35b8ffaa17	docs/vmagent.md: clarify `-promscrape.suppressDuplicateScrapeTargetErrors` command-line flag usage	2020-10-08 19:24:05 +03:00
Aliaksandr Valialkin	85a1ab3edd	CHANGELOG.md: mention features from `71ea4935de`	2020-10-08 19:14:45 +03:00
Aliaksandr Valialkin	0d44e371f3	lib/promscrape: add `-promscrape.suppressDuplicateScrapeTargetErrors` command-line flag in order to suppress `duplicate scrape target` errors Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin	ad0950f630	lib/backup/actions: improve logging to be more clear to humans	2020-10-08 14:23:00 +03:00
Aliaksandr Valialkin	f9f8e4a39c	app/vmalert: do not pring description for all the flags on config errors The description is too big to consume by human and it just distracts humans.	2020-10-08 13:35:46 +03:00
Aliaksandr Valialkin	7786ed95b5	vendor: `make vendor-update`	2020-10-08 11:52:43 +03:00
Aliaksandr Valialkin	3e2bf32872	vendor: update github.com/VictoriaMetrics/metricsql from v0.7.0 to v0.7.1	2020-10-08 11:47:23 +03:00
Aliaksandr Valialkin	ef9280201c	CHANGELOG.md: mentioned about the added optimization that adds missing filters to binary operands	2020-10-07 21:23:16 +03:00
Aliaksandr Valialkin	f6ee6efc34	app/vmselect/promql: add missing label filters to binary operands before query execution This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization See also https://github.com/cortexproject/cortex/issues/3253	2020-10-07 21:17:11 +03:00
Aliaksandr Valialkin	b8024db965	CHANGELOG.md: mention about `-finalMergeDelay` comand-line flag	2020-10-07 18:52:35 +03:00
Dmitry Shihovtsev	aec863e70b	Fix typos in the vmalert datasource (#814 ) * Fix typos in the vmalert datasource * Fix typo in the vmalert datasource test	2020-10-07 18:00:29 +03:00
Artem Navoiev	9f1656145b	update go action	2020-10-07 17:48:38 +03:00
Aliaksandr Valialkin	b51fa16177	app/vmstorage: add `-finalMergeDelay` command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin	5d5076c4a2	docs/CaseStudies.md: actualize Wix numbers	2020-10-06 16:09:30 +03:00
Aliaksandr Valialkin	b4356550fd	CHANGELOG.md: add missing link to an issue about OpenStack service discovery - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-06 15:37:26 +03:00
Aliaksandr Valialkin	c4d309aa41	app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page	2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin	abfd3a8fab	app/{vminsert,vmselect,vmstorage}: add a link to https://victoriametrics.github.io/Cluster-VictoriaMetrics.html from main page of every cluster component	2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin	70bb5624ad	docs/Articles.md: add https://medium.com/@VictoriaMetrics/anomaly-detection-in-victoriametrics-9528538786a7	2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin	1602f55794	CHANGELOG.md: cut v1.43.0 release	2020-10-06 14:29:06 +03:00
Aliaksandr Valialkin	49451854ef	CHANGELOG.md: add missing entries for upcoming release	2020-10-06 12:04:52 +03:00
Aliaksandr Valialkin	54ff78c6c9	lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does Such timestamps are rounded to seconds like Carbon does. See `b0ba62a62d/lib/carbon/protocols.py (L197)`	2020-10-06 11:38:35 +03:00
Aliaksandr Valialkin	1da41177a8	lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human	2020-10-06 11:20:36 +03:00
Aliaksandr Valialkin	97b836a6f4	lib/fs: fix GOOS=openbsd build by adding `fadviseSequentialRead` implementation. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785	2020-10-05 23:32:28 +03:00
Aliaksandr Valialkin	812c670d60	vendor: `make vendor-update`	2020-10-05 23:19:49 +03:00
Aliaksandr Valialkin	4e8fad94a5	.github/workflows: switch Go version from v1.14 to v1.15	2020-10-05 22:00:47 +03:00
Aliaksandr Valialkin	50316070d6	lib/promscrape/discovery/openstack: code prettifying after `cbe3cf683b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-05 18:12:31 +03:00
Nikolay Khramchikhin	b4c77fc6d2	Adds openstack sd (#811 ) * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin	5f15c52d21	docs: `make docs-sync` after `cbe3cf683b`	2020-10-05 16:48:25 +03:00
Aliaksandr Valialkin	7505e4f390	lib/promrelabel: make a copy of label with new name for `action: labelmap` in the same way as Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812	2020-10-05 16:20:22 +03:00
Aliaksandr Valialkin	9a7c863bd8	lib/protoparser/influx: add `-influx.maxLineSize` command-line flag for configuring the maximum size for a single Influx line during parsing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807	2020-10-05 15:19:11 +03:00
Aliaksandr Valialkin	11ddb79aeb	lib/decimal: add tests for negative values passed to maxUpExponent	2020-10-05 14:57:03 +03:00
Aliaksandr Valialkin	d448e07546	lib/decimal: properly calibrate scale for blocks with Inf values See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805	2020-10-05 14:52:59 +03:00
Aliaksandr Valialkin	3e0c473cc9	app/vmselect/promql: fill gaps on graphs for `range_` and `running_` functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806	2020-10-02 14:02:47 +03:00
Aliaksandr Valialkin	fd7dd5064a	lib/storage: code cleanup after `10f2eedee0` Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:12:04 +03:00
Aliaksandr Valialkin	3ad7566a87	lib/storage: imrpove cache effectiveness for time series ids matching the given filters Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day. This should reduce CPU usage in the following cases: * when querying recently added data with small churn rate for time series * when querying historical data	2020-10-01 14:39:46 +03:00
Aliaksandr Valialkin	40df42e1e5	docs: `make docs-sync`	2020-09-30 09:51:16 +03:00
Roman Khavronenko	368b890e11	vmalert: make maxIdleConnections configurable for datasource HTTP client (#797 ) Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795	2020-09-30 09:51:14 +03:00
Aliaksandr Valialkin	af7a0f7aff	vendor: `make vendor-update`	2020-09-30 08:59:53 +03:00
Aliaksandr Valialkin	10b6982ef8	docs/Release-Guide.md: mention that CHANGELOG.md must be updated before release	2020-09-30 08:53:27 +03:00
Aliaksandr Valialkin	ebb1f9ebb6	CHANGELOG.md: release v1.42.0	2020-09-30 08:45:10 +03:00
Aliaksandr Valialkin	081c5d2c74	CHANGELOG.md: initial commit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/788	2020-09-30 00:12:23 +03:00
Aliaksandr Valialkin	7c2e4e267a	lib/storage: allow set values higher than 1 for `vm_merge_need_free_disk_space` if there are multiple partitions with deferred merges due to disk space shortage	2020-09-29 22:53:34 +03:00
Aliaksandr Valialkin	536aa8779a	app/vmstorage: rename `vm_{big\|small}_merge_need_free_disk_space` to `vm_merge_need_free_disk_space` This simplifies alerting.	2020-09-29 22:53:33 +03:00
Aliaksandr Valialkin	097a4c10dd	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin	a4361a4c07	docs/Single-server-VictoriaMetrics.md: typo fix	2020-09-29 20:30:26 +03:00
Aliaksandr Valialkin	7ae0ab976a	docs/Single-server-VictoriaMetrics.md: typo fix: compations -> compactions	2020-09-29 20:27:37 +03:00
Aliaksandr Valialkin	5dca7bbe85	app/vmagent/remotewrite: do not show `-remoteWrite.url` in logs if `-remoteWrite.showURL` isn't set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin	61f4801b93	app/vmselect/graphite: do not substitute `path` and `path.` with `path..` in `/metrics/find/?format=completer` output	2020-09-29 18:03:45 +03:00
Aliaksandr Valialkin	5e998e597a	lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309	2020-09-29 13:55:53 +03:00
Aliaksandr Valialkin	f4d74ccfec	docs/{vmbackup,vmrestore}: formatting fixes	2020-09-29 13:19:13 +03:00
Aliaksandr Valialkin	5179df7082	docs/vmbackup.md: make docs about minio config more prominent	2020-09-29 13:16:34 +03:00
Aliaksandr Valialkin	338a53ccf9	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:37 +03:00
Aliaksandr Valialkin	79b00f7b6b	docs: improve readability a bit	2020-09-29 13:05:05 +03:00
Nikolay Khramchikhin	d77f56fd97	update vmbackup/vmrestore README usage (#794 ) * update vmbackup/vmrestore README usage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/381 with minio and configuration file examples. * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * added backup/restore docs changes * added example for relabelConfig flag * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-29 13:05:04 +03:00
Aliaksandr Valialkin	ef416c72c2	lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm	2020-09-29 12:42:25 +03:00
Aliaksandr Valialkin	0b0259c42c	lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic	2020-09-29 12:29:50 +03:00
Aliaksandr Valialkin	8f25206d8c	lib/protoparser/prometheus: fix TestParseStream after `124f78857b`	2020-09-29 12:12:33 +03:00
Aliaksandr Valialkin	0d082cdf53	app/vmselect/netstorage: remove unused code	2020-09-29 11:55:08 +03:00
Aliaksandr Valialkin	c47972d843	app/vmselect/prometheus: check for errors returned from bufferedwriter.Write This makes `make errcheck` happy	2020-09-29 11:36:54 +03:00
Aliaksandr Valialkin	e66f7edfc9	app/vmselect/graphite: properly handle case when `/metrics/find` finds both leaf and node for the given `query=prefix.*` In this case only node must be returned with stripped dot in the end of id as carbonapi does	2020-09-29 11:29:38 +03:00
Aliaksandr Valialkin	ffa6581c46	app/vminsert: refresh the list of healthy storage nodes only if the the row cannot be sent to destination storage node Previously the list had been generated for each rerouted row. This could consume additional CPU time during rerouting, which could lead to rerouting slowdown. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791	2020-09-29 01:29:24 +03:00
Aliaksandr Valialkin	1e7452e501	.github/workflows: verify builds for vmagent, vmalert, vmbackup and vmrestore	2020-09-29 00:47:54 +03:00
Aliaksandr Valialkin	f68bf12a84	.github/workflows: verify that VictoriaMetrics can be built for GOOS=openbsd	2020-09-29 00:45:38 +03:00
Aliaksandr Valialkin	81cdf2fa14	lib/{fs,filestream}: small consistency-related updates after `cc90a548b1`	2020-09-29 00:43:20 +03:00
Nikolay Khramchikhin	658a05ef0f	added openbsd implementations (#790 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:43:19 +03:00
Aliaksandr Valialkin	bc37f1cbec	app/vminsert: do not pollute logs with repated `cannot dial storageNode` errors Log only the first error per -storageNode	2020-09-29 00:20:32 +03:00
Aliaksandr Valialkin	0548f0c5c8	vendor: `make vendor-update`	2020-09-28 21:59:20 +03:00
Aliaksandr Valialkin	2ee0dc27a6	app/vmstorage: parallelize data processing obtained from a single connection from vminsert Previously vmstorage could use only a single CPU core for data processing from a single connection from vminsert. Now all the CPU cores can be used for data processing from a single connection from vminsert. This should improve the maximum data ingestion performance for a single vminsert->vmstorage connection.	2020-09-28 21:41:16 +03:00
Aliaksandr Valialkin	9d123eb22a	app/vminsert: remove useless delays when sending data to vmstorage This improves the maximum data ingestion performance for cluster VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791	2020-09-28 21:41:15 +03:00
Aliaksandr Valialkin	1481d6d8ff	lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer	2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin	8df33bd5c1	app/{vminsert,vmagent}: improve data ingestion speed over a single connection Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin	7072db75cb	lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server This should reduce syscall overhead when reading big amounts of data	2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin	6d8c23fdbd	app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics	2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin	db14f22fc0	app/vmselect: stop `/api/v1/export/*` execution if client disconnects	2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin	aadbd014ff	all: add native format for data export and import The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin	00ec2b7189	lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.	2020-09-26 04:22:06 +03:00
Aliaksandr Valialkin	e83947a882	app/vminsert: code prettifying	2020-09-26 04:16:14 +03:00
Aliaksandr Valialkin	8fc9b77496	app/vmagent: reduce memory usage when importing data via /api/v1/import Previously vmagent could use big amounts of RAM when each ingested JSON line contained many samples.	2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin	973df09686	app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during `/api/v1/series` calls	2020-09-24 20:44:15 +03:00
Aliaksandr Valialkin	533bf76a12	lib/storage: correctly use maxBlockSize in various checks Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.	2020-09-24 18:13:15 +03:00
Aliaksandr Valialkin	b8bce348c5	app/vmselect/promql: properly limit implicitly set rollup window to `-search.maxStalenessInterval` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784	2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin	543f3aea97	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin	265d892a8c	vendor: `make vendor-update`	2020-09-23 14:23:51 +03:00
Aliaksandr Valialkin	8627365b48	app/vmselect/prometheus: code cleanup after `3ba507000c`	2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin	1fce79518a	app/vmselect/prometheus: return timestamps from `/api/v1/query`, which match the `time` query arg Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin	90d2549428	lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue	2020-09-23 02:17:53 +03:00
Aliaksandr Valialkin	0468cdf33e	app/vmselect/netstorage: properly pre-allocate space for sbs	2020-09-22 23:51:01 +03:00
Aliaksandr Valialkin	89d652b583	lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728	2020-09-22 23:27:26 +03:00
Aliaksandr Valialkin	4ebd2fa560	vendor: `make vendor-update`	2020-09-22 17:08:54 +03:00
Aliaksandr Valialkin	00ff01f766	docs/Single-server-VictoriaMetrics.md: VictoriaMetrics properly stores Inf values after `26115891db`	2020-09-22 17:03:16 +03:00
Aliaksandr Valialkin	69eb9783e6	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:21:42 +03:00
Aliaksandr Valialkin	31e341371b	lib/storage: code prettifying after `be5e1222f3` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-09-22 00:42:20 +03:00
faceair	ad41e39350	add filter to getMetricIDs (#783 ) * add getMetricIDs filter * check nil filter before use	2020-09-22 00:42:19 +03:00
Aliaksandr Valialkin	ed473c94ff	docs/vmagent.md: typo fix	2020-09-21 21:49:08 +03:00
Aliaksandr Valialkin	bedc971398	docs: `make docs-sync`	2020-09-21 21:48:05 +03:00
Aliaksandr Valialkin	e564725641	app/vmselect/searchutils: fixed tests after `2eb72e09ab`	2020-09-21 21:31:28 +03:00
Aliaksandr Valialkin	07c6226334	app/vmselect: use `time` value rounded to seconds if it isnt passed to `/api/v1/query` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:46 +03:00
Aliaksandr Valialkin	d32c3f747c	lib/logger: add `-loggerDisableTimestamps` command-line flag for disabling timestamps in logs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778	2020-09-21 19:25:50 +03:00
Aliaksandr Valialkin	f961838290	lib/promscrape/discovery/ec2: code prettifying after `312fead9a2`	2020-09-21 18:44:04 +03:00
Nikolay Khramchikhin	0069353d5e	Add improvements to ec2_sd_discovery (#775 ) * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin	8cd89cb847	app/vmalert: remove unneeded UTC() call UTC() doesn't change the underlying timestamp, so the call isn't needed here	2020-09-21 15:56:48 +03:00
Roman Khavronenko	d111969d39	vmalert: add support for `datasource.lookback` flag (#779 ) New datasource flag `datasource.lookback` defines how far to look into past when evaluating queries. Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668	2020-09-21 15:56:47 +03:00
Aliaksandr Valialkin	a9321f6a60	lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges	2020-09-21 15:51:26 +03:00
Roman Khavronenko	0042b0f307	vmalert: fix the typo in error message (#782 ) The error will be always nil so no sense in printing it.	2020-09-21 11:36:09 +03:00
Aleksey Shirokih	a7e17f14f5	dashboard: add account id to datapoints ingestion rate (#772 )	2020-09-19 22:00:16 +01:00
Aliaksandr Valialkin	604e8f6114	lib/decimal: optimize maxUpExponent() by eliminating division from hot path	2020-09-19 13:49:43 +03:00
Aliaksandr Valialkin	df547bf345	lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush	2020-09-19 12:51:46 +03:00
Aliaksandr Valialkin	778ea183ca	lib/decimal: properly store Inf values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-18 19:08:53 +03:00
Aliaksandr Valialkin	1c07d7bee7	app/vmagent: increase default value for `-remoteWrite.queues` from 1 to 4, since it has been appeared that many users hit this limit	2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin	5bd30171b3	vendor: update github.com/valyala/quicktemplate from v1.6.2 to v1.6.3	2020-09-18 13:10:55 +03:00
Aliaksandr Valialkin	adb4d3b75c	lib/persistentqueue: flush data to disk every second Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-18 13:06:03 +03:00
Aliaksandr Valialkin	8fd791d399	lib/promscrape: avoid copying response body when scraping targets. This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:06:00 +03:00
Aliaksandr Valialkin	e25b90849f	vendor: udpate github.com/VictoriaMetrics/fasthttp from v1.0.5 to v1.0.7	2020-09-18 12:20:48 +03:00
Aliaksandr Valialkin	a1bebb660c	app/vmselect/graphite: return proper results `/metrics/find?query=foo.*.bar` according to Graphite Metrics API	2020-09-18 11:53:28 +03:00
Aliaksandr Valialkin	9b15b11f74	app/vmstorage: added `-forceMergeAuthKey` command-line flag for protecting `/internal/force_merge` endpoint	2020-09-17 14:24:20 +03:00
Aliaksandr Valialkin	d96858b921	lib/storage: add `/internal/force_merge` handler for running forced compactions on historical per-month partitions This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin	3abbb38254	lib/{mergeset,storage}: compare errors with `errors.Is()`	2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin	ddb3519e17	lib/{mergeset,storage}: code prettifying	2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin	bf826dd828	lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions	2020-09-17 01:49:08 +03:00
Aliaksandr Valialkin	7429fb2301	vendor: `make vendor-update`	2020-09-17 01:44:16 +03:00
Aliaksandr Valialkin	2084496462	docs/Single-server-VictoriaMetrics.md: document that /api/v1/series/count may count delete time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/770	2020-09-17 01:39:09 +03:00
Aliaksandr Valialkin	406f4fe445	app/vmagent: substitute `-remoteWrite.url` with `secret-url` value in logs, since it may contain sensitive info such as passwords or auth tokens Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin	9793008734	app/vmselect: add -search.storageTimeout command-line flag for limiting the maximum duration of query execution per each -storageNode See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:33:47 +03:00
Aliaksandr Valialkin	9705ac5d7a	lib/persistentqueue: code simplification after `d455764a6f`	2020-09-16 21:14:01 +03:00
Aliaksandr Valialkin	a9205fe308	app/vmselect: prevent from closing connection to vmstorage on query timeout by setting +2 secs deadline on connection comparing to query deadline Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:14:00 +03:00
Aliaksandr Valialkin	eee6f1e56d	lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset) The strategy is: - Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown. - Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data. - Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown. - Skipping of broken chunk file if it cannot be read. - Fsyncing finalized chunk files. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-16 18:13:24 +03:00
Aliaksandr Valialkin	6ce52e3702	lib/protoparser/vmimport: add more testcases for invalid timestamps and values Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:21:53 +03:00
Aliaksandr Valialkin	cd87ca303f	lib/protoparser: report more errors for incorrect timestamps and/or values Previously certain errors in timestamps and/or values could be silently skipped, which could lead to samples with zero values stored in the database. Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin	5c4e111b43	lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99	2020-09-16 02:16:13 +03:00
Aliaksandr Valialkin	1587f83fa0	app/vmselect/netstorage: typo fix after `03dfccfbed`	2020-09-16 00:10:33 +03:00
Aliaksandr Valialkin	2a86e2fb98	vendor: update github.com/valyala/fastjson from v1.5.4 to v1.5.5 This should properly parse `+Inf` values when importing JSON lines via `/api/v1/import` Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 00:09:05 +03:00
Aliaksandr Valialkin	d39a985d6b	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics ignores NaN and Inf values during data ingestion Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-15 23:41:10 +03:00
Aliaksandr Valialkin	03dfccfbed	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:09 +03:00
Aliaksandr Valialkin	27cd5555e6	app/vmselect/netstorage: mention that RunParallel or Cancel must be called on the returned results from ProcessSearchQuery	2020-09-15 20:39:43 +03:00
Aliaksandr Valialkin	1f0c0b0f6b	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:25:34 +03:00
Aliaksandr Valialkin	0c1c1b79ba	lib/promscrape: add a link to troubleshooting docs to error message when duplicate scrape target with identical labels is skipped	2020-09-15 14:16:20 +03:00
Aliaksandr Valialkin	5f04b00b2d	docs/Articles.md: add a link to https://medium.com/miro-engineering/prometheus-high-availability-and-fault-tolerance-strategy-long-term-storage-with-victoriametrics-82f6f3f0409e	2020-09-15 12:29:44 +03:00
Aliaksandr Valialkin	ca08161b54	lib/promscrape: typo fix	2020-09-12 00:14:15 +03:00
Roman Khavronenko	e2b31590e6	vmalert: add Group name as label to generated alerts and timeseries (#761 ) Solves #611	2020-09-11 23:41:12 +03:00
Roman Khavronenko	16e0bb496e	vmalert: update groups on config reload only if changes detected (#759 ) On config reload event `vmalert` reloads configuration for every group. While it works for simple configurations, the more complex and heavy installations may suffer from frequent config reloads. The change introduces the `checksum` field for every group and is set to md5 hash of yaml configuration. The checksum will change if on any change to group definition like rules order or annotation change. Comparing the `checksum` field on config reload event helps to detect if group should be updated. The groups update is now done concurrently, so reload duration will be limited by the slowest group now. Partially solves #691 by improving config reload speed.	2020-09-11 23:41:12 +03:00
Aliaksandr Valialkin	e53235ac5c	lib/promscrape: do not reset the remaining rows when pushing a part of data to remote storage during big scrapes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753 Thanks to @PerGon and @clmssz for help with debugging.	2020-09-11 23:38:17 +03:00
Aliaksandr Valialkin	b776a93608	app/vmselect/promql: support composite durations like Prometheus 2.21 does The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin	114cf24b43	lib/promscrape/discovery/dns: add `__meta_dns_srv_record_target` and `__meta_dns_srv_record_port` labels This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases and https://github.com/prometheus/prometheus/pull/7678 .	2020-09-11 21:35:39 +03:00
Aliaksandr Valialkin	6382e8081a	app/vmagent: allow setting multiple identical `-remoteWrite.url` values This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755	2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin	5cf5a0e8c4	lib/protoparser/common: do not read request body when parsing `timestamp` query arg This was preventing from reading data via /api/v1/prometheus/import . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin	81c05f669b	lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin	8bd5aa3516	lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus This improves consistency with timestamps in Prometheus text exposition format	2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin	d3ad0d365e	app/vmselect: move Deadline from netstorage to searchutils This removes dependency on netstorage from searchutils.	2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin	58d3b82ae5	app/{vminsert,vmagent}: allow passing timestamp via `timestamp` query arg when ingesting data to `/api/v1/import/prometheus` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:28:31 +03:00
Nikolay Khramchikhin	af994562c8	Added endpointslices discovery to k8s api (#760 ) This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21. See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1 * Added endpointslices discovery to k8s api Started from 1.17 k8s version endpointslices is beta, it allows to query k8s api for endpoints more efficient. It presents at scrape_config.yaml as separate role for kubernetes_sd_config. kubernetes_sd_config: - role: endpointslices * fixed typos, changed EndpointConditions signature - with values instead of pointers	2020-09-11 12:24:50 +03:00
Aliaksandr Valialkin	579c20756a	app/vmselect: substitute inf values at smooth_exponential with the previous values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin	d67e6d3d2e	app/vmselect: skip infinite values when calculating smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:57:53 +03:00
John Belmonte	a72a2566d7	fix typo on outliersk() doc (#758 )	2020-09-11 01:59:55 +03:00
Aliaksandr Valialkin	06427a184f	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:59:52 +03:00
Aliaksandr Valialkin	f307e6f432	app/vmselect: initial implementation of Graphite Metrics API See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin	87f916a2fb	vendor: `make vendor-update`	2020-09-10 09:50:00 +03:00
Aliaksandr Valialkin	5afe69407d	deployment/docker: update Go builder from v1.15.1 to v1.15.2 This fixes the following issues in Go runtime - see https://github.com/golang/go/issues?q=milestone%3AGo1.15.2+label%3ACherryPickApproved	2020-09-10 09:38:10 +03:00
Aliaksandr Valialkin	f5cb213ef9	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin	475698d2ad	docs: sync docs for vmalert, vmauth, vmbackup and vmrestore	2020-09-09 21:10:48 +03:00
Aliaksandr Valialkin	d72fd86488	docs/Articles.md: add links to recently published third-party articles and talks about VictoriaMetrics	2020-09-09 20:15:44 +03:00
Aliaksandr Valialkin	5a82f645e4	docs/Single-server-VictoriaMetrics.md: typo fix	2020-09-09 01:22:18 +03:00
Aliaksandr Valialkin	95ad657a1c	docs/Single-server-VictoriaMetrics.md: typo fix	2020-09-09 01:00:05 +03:00
Aliaksandr Valialkin	5ab57f916b	docs/vmagent.md: clarified the case when `-remoteWrite.queues` must be tuned Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745	2020-09-08 20:15:49 +03:00
Aliaksandr Valialkin	e5c8377212	lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem	2020-09-08 19:57:20 +03:00
Aliaksandr Valialkin	df6519c190	vendor: `make vendor-update`	2020-09-08 15:18:59 +03:00
Aliaksandr Valialkin	f3a79abfb4	app/vmselect/promql: go fmt	2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin	4f06eed1c1	app/vmselect/promql: adjust `integrate()` calculations to be more similar to calculations from InfluxDB: attempt #2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin	0d0b606455	app/vmselect/promql: adjust `integrate()` calculations to be more similar to calculations from InfluxDB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin	db91045348	app/vmselect/promql: increase floating point calculations accuracy by dividing by `1e3` instead of multiplying by `1e-3`	2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin	13dd915302	docs/Single-server-VictoriaMetrics.md: `make docs-sync`	2020-09-07 21:58:48 +03:00
Nikolay Khramchikhin	fb356c434b	Changed s3 configProfile flag default, (#749 ) aws sdk has complicated logic for chosing profile name and we shouldn't set it to `default` value. It leads to bugs and improper configuration. Set it to empty value by default is safe. It will be automatically set to `default` by sdk.	2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin	5cb8c82fe5	docs/Single-server-VictoriaMetrics.md: sync with README.md	2020-09-04 03:30:43 +03:00
Aliaksandr Valialkin	38462bd95e	docs/Single-server-VictoriaMetrics.md: updates according to review comments at `fe98ba5a60`	2020-09-04 03:08:20 +03:00
Aliaksandr Valialkin	804304c365	app/vmselect: add missing deletion for temporary files on partial responses when `-search.denyPartialResponse=true`	2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin	d2f6f96e4a	lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699	2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin	281d715060	lib/httpserver: add `-http.connTimeout` command-line flag for limiting the lifetime for incoming http connections This can be useful for balancing incoming connections among multiple services.	2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin	fcde009e11	vendor: update github.com/VictoriaMetrics/metricsql from v0.4.3 to v0.5.1 The new version of the package supports binary operations on string literals: * "foo" + "bar" => "foobar" * "foo" == "bar" => NaN * "foo" == "foo" => 1 * "foo" >bool "bar" => 1 * "foo" < "bar" => NaN Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/717	2020-09-03 16:33:23 +03:00
Aliaksandr Valialkin	478d8f8393	app/vmselect/promql: add `count_le_over_time(m[d], le)` and `count_gt_over_time(m[d], gt)` functions These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin	c3a2d4ee6f	vendor: update github.com/VictoriaMetrics/metricsql from v0.4.1 to v0.4.2 The new version of this package properly supports escaped identifiers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/743	2020-09-03 14:51:06 +03:00
Aliaksandr Valialkin	3490160fd0	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin	7ab7ae79c7	docs/Single-server-VictoriaMetrics.md: mention that `/api/v1/series` returns series for the last 5 minutes if `start` query arg is missing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/735	2020-09-03 12:39:31 +03:00
Aliaksandr Valialkin	a3cdef6b06	app/vmagent: properly flush big blocks of data Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741 Thanks to @IceRain00 for the investigation and initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742	2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin	de216bab41	app/vmagent: fix data race when accessing writeRequest.lastFlushTime	2020-09-03 12:12:09 +03:00
Nikolay Khramchikhin	80a9dc79fe	changed vmalert behaviour (#738 ) * VMAlert start with empty rules dir There are some applications (operator for instance), that generates alerts configuration at runtime and vmalert must start correctly without rules to support this behaviour. Later application will add rules files and send SIGHUP to vmalert, which will trigger reading rules files and start rules exectuion. Removing rules files with SIGHUP signal must stop rules execution and vmalert will wait for new rules. * imports sorted * added test cases for empty rules, removed blank line * fixed imports conflict * updated tests	2020-09-03 11:07:40 +03:00
Aliaksandr Valialkin	4388c6cad1	docs/Single-server-VictoriaMetrics.md: add missing link to Prometheus text exposition format	2020-09-03 01:14:33 +03:00
Aliaksandr Valialkin	7ac10ee978	app/vmalert: imrovements over `3f932c2db1`	2020-09-03 01:14:30 +03:00
DexterZhang	85f49ad439	feat: spread load of rule evaluation by group when starting new groups (#724 ) * feat: spread load of rule evaluation by group when starting new groups * review: reduce the resulting diff. * Update app/vmalert/group.go Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-03 01:14:26 +03:00
Aliaksandr Valialkin	4fa97430d7	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin	95ce89e7d7	lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling The previous basis on `cap(sw.labels)` doesn't work anymore after `7785869ccc` , because `sw.labels` may be reset multiple times when processing big number of rows.	2020-09-02 18:46:55 +03:00
Roman Khavronenko	432c0383db	Single dashboards update (#736 ) * dashboard: rename var `datasource` to `ds` for consistency reason Dasbhoards for cluster version or vmagent operate with datasource variable named `ds`. For consistency sake we rename this variable in single node version as well. * dashboard: add instance variable picker See dashboard reviews here https://grafana.com/grafana/dashboards/10229/reviews * dashboard: limit number of buckets in histogram to 12 for vmagent dashboard * dashboard: bump version requirement in description for single version * dashboard: drop extra series override for single version * dashboard: set Y-min to zero for most of panels in vmagent dashboard	2020-09-02 15:18:29 +03:00
Roman Khavronenko	801a26340f	dashboard: set Y-min to zero for most of panels in cluster dashboard (#737 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-09-02 15:15:13 +03:00
Aliaksandr Valialkin	63052f80fb	deployment/docker: update Go builder from v1.15.0 to v1.15.1	2020-09-02 15:10:07 +03:00
John Belmonte	5ea6f86dd8	use Y-min 0 on Grafana dashboard graphs (#732 )	2020-09-02 15:06:17 +03:00
Aliaksandr Valialkin	74ba0a6271	vendor: `make vendor-update`	2020-09-01 17:43:27 +03:00
Aliaksandr Valialkin	bc1ca4b20b	lib/httpserver: add `-http.idleConnTimeout` command-line flag for tuning the timeout for incoming idle http connections	2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin	a01c56104a	lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics This has been broken at `7785869ccc`	2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin	deff8d419a	lib/promscrape: reduce memory usage when scraping targets with millions of metrics This should help when scraping /federate endpoints from Prometheus instances, which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/	2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin	fe08b1eb26	app/vminsert: improve error message when the data cannot be sent to vmstorage - log reroutedBR buffer size This should improve debuggability for improperly configured cluster	2020-08-31 17:51:44 +03:00
Aliaksandr Valialkin	6f9c1bc078	app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url	2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin	3b1ecac04b	app/vmagent: apply sane limits to `-remoteWrite.queues` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707	2020-08-30 21:25:51 +03:00
Aliaksandr Valialkin	49140edd41	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics accepts relative times at `time`, `start` and `end` query args	2020-08-28 10:14:33 +03:00
Aliaksandr Valialkin	7e74cf4d71	docs/vmalert.md: sync with app/vmalert/README.md via `make docs-update`	2020-08-28 09:52:10 +03:00
Aliaksandr Valialkin	815dffabed	docs/Articles.md: add a link to https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-filtering-and-modifying-time-series-6d40cea4bf21	2020-08-28 09:52:09 +03:00
Aliaksandr Valialkin	5f2277624a	lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy	2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin	45e770ed20	lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.	2020-08-28 09:49:22 +03:00
Roman Khavronenko	08b76cb26f	vmalert: update `-rule` flag description to enforce quotes using (#709 ) Description for `-rule` flag uses as example specific chars like asterisks which could be interpreted wrong by different shells. To avoid this, description now contains quoted flag values. See also #708	2020-08-28 09:46:35 +03:00
Roman Khavronenko	34ef10fbcc	lib/flagutil: avoid int overflow for arch 386 (#710 ) Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32, whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with `int overflow` error.	2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin	9a77ae9d1c	lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels Previously targets with big number of metrics and/or labels could generated too big buffers, which then could be re-used when scraping targets with small number of metrics. This resulted in memory waste. Now big buffers are used only for targets with big number of metrics / labels, while small buffers are used for targets with small number of metrics / labels.	2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin	3ea6444219	lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket This should reduce the number of resizings for the returned byte buffers.	2020-08-16 22:13:38 +03:00
Roman Khavronenko	4b89da9463	lib/decimal: rename `significant decimal digits` to `significant figures` (#698 ) The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using `KB`, `MB`, `GB`, `KiB`, `MiB` and `GiB` suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	1b5467f7fd	lib/memory: improve log message about the memory allowed to use by VictoriaMetrics	2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin	d9f7ea1c6e	lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data The OS should return any buffered data in the stream without the need to set the read timeout. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin	98217e4c40	vendor: upgrade github.com/valyala/gozstd from v1.7.1 to v1.8.3	2020-08-15 15:11:50 +03:00
Aliaksandr Valialkin	008a769434	vendor: downgrade github.com/valyala/gozstd from v1.8.1 to v1.7.1 until https://github.com/facebook/zstd/issues/2222 is fixed	2020-08-15 14:51:16 +03:00
Aliaksandr Valialkin	edf3ee2a9b	lib: dump compressed block contents on error during decompression This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222	2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin	7fb942308c	vendor: update github.com/valyala/gozstd from v1.7.0 to v1.8.1	2020-08-15 13:46:34 +03:00
Aliaksandr Valialkin	6e863376f7	lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty This should reduce memory allocations and copying when the byte slice is growing.	2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin	285665e93b	app/vmselect/promql: allow passing multiple args to aggregate functions such as `avg(q1, q2, q3)`	2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin	a2021d0dde	docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion	2020-08-14 20:48:17 +03:00
Aliaksandr Valialkin	3efa4e4e1c	lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin	c6b0547847	lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin	9d79a3a99d	lib/memory: add `-memory.allowedBytes` command-line flag for setting absolute memory limit for VictoriaMetrics caches	2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin	e7c0b2ca56	docs: update docs	2020-08-14 19:14:46 +03:00
Aliaksandr Valialkin	b996280c65	app/{vminsert,vmagent}: improve documentation for `-influxListenAddr` command-line flag	2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin	c82a485cf6	lib/protoparser/prometheus: typo fix in error message	2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin	f4c90449dc	deployment/dm: update images	2020-08-14 02:27:55 +03:00
Aliaksandr Valialkin	0a34f56b39	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.4 to v1.0.5	2020-08-14 02:19:56 +03:00
Aliaksandr Valialkin	d615ae81e5	vendor: update github.com/klauspost/compress from v1.10.10 to v1.10.11	2020-08-14 02:16:39 +03:00
Aliaksandr Valialkin	9e67343756	lib/promscrape: use a hint on body length instead of body capacity This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin	b4119bb51e	lib/promscrape: reduce memory usage when scraping big number of targets Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin	1724cc241e	lib/promscrape: properly retry requests on `the server closed connection before returning the first response byte` error during service discover API calls and target scrapes	2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin	60c7397be5	all: support `%{ENV_VAR}` placeholders in yaml configs in all the vm* components Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin	aa7f1a9d8f	deployment/docker: update Go builder from Go1.14.7 to Go1.15.0	2020-08-13 15:53:39 +03:00
Aliaksandr Valialkin	c5d50a5940	docs/Cluster-VictoriaMetrics.md: mention about Kubernetes operator	2020-08-12 21:15:01 +03:00
Aliaksandr Valialkin	801a0241fc	docs/Single-server-VictoriaMetrics.md: mention helm charts, k8s operator and vmctl tool in `Integrations` chapter	2020-08-12 21:12:45 +03:00
Aliaksandr Valialkin	f20032dbb5	docs/Articles.md: added https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043	2020-08-12 21:03:25 +03:00
Aliaksandr Valialkin	6721e47ae9	app: respect CPU limits set via cgroups Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin	59d95961b8	lib/protoparser: clarify that the string passed to `Unmarshal()` function must remain available when the parsed rows are in use	2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin	315b752245	docs/Single-server-VictoriaMetrics.md: mention that it is safe to skip multiple versions during the upgrade	2020-08-11 14:22:24 +03:00
Aliaksandr Valialkin	62b6e54622	app/vmselect: reduce memory usage when exporting time series with big number of samples via `/api/v1/export` if `max_rows_per_line` is set to non-zero value Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin	890cfe5b61	lib/protoparser/influx: accept `precision=us` and `precision=µ` according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint	2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin	e3439a6cd0	lib/promscrape: optimize per-metric hash calculations This increases vmagent performance by up to 10% when scraping big number of metrics	2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin	c9f5c5623f	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin	4ce1368e4b	lib/storage: mention time range used in the query that led to error message This should improve detecting slow queries with too big time ranges	2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin	f92255e803	lib/storage: mention tag filters used in the query that led to error message This should improve detecting invalid or heavy queries that lead to errors.	2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin	b3d4ff7ee2	app/vmstorage: improve error logging when the request times out	2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin	e3999ac010	lib/promscrape: show real timestamp and real duration for the scape on `/targets` page Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp	2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin	43830b1699	vendor: `make vendor-update`	2020-08-09 15:14:09 +03:00
Aliaksandr Valialkin	c09c881264	lib/promscrape: make errcheck happy	2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin	2dfb42a8b4	lib/promscrape: export `scrape_samples_added` per-target metric like Prometheus does This metric may be useful for detecting targets with high churn rate for the exported metrics. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683	2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin	fd9f1463df	lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails this is expected condition, so it is better to use WARN log level for it	2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin	d4be3efc60	lib/promscrape: add a test for scrape config for blackbox exporter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684	2020-08-09 12:03:51 +03:00
Roman Khavronenko	78afc61896	app/vmalert: extend metrics set exported by `vmalert` #573 (#654 ) * app/vmalert: extend metrics set exported by `vmalert` #573 New metrics were added to improve observability: + vmalert_alerts_pending{alertname, group} - number of pending alerts per group per alert; + vmalert_alerts_acitve{alertname, group} - number of active alerts per group per alert; + vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error during prev execution, is 0 if no errors happened; + vmalert_recording_rules_error{recording, group} - is 1 if recording rule ended up with error during prev execution, is 0 if no errors happened; * vmalert_iteration_total{group, file} - now contains group and file name labels. This should improve control over specific groups; * vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups; Some collisions for alerts and recording rules are possible, because neither group name nor alert/recording rule name are unique for compatibility reasons. Commit contains list of TODOs for Unregistering metrics since groups and rules are ephemeral and could be removed without application restart. In order to unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13 * app/vmalert: extend metrics set exported by `vmalert` #573 The changes are following: * add an ID label to rules metrics, since `name` collisions within one group is a common case - see the k8s example alerts; * supports metrics unregistering on rule updates. Consider the case when one rule was added or removed from the group, or the whole group was added or removed. The change depends on https://github.com/VictoriaMetrics/metrics/pull/16 where race condition for Unregister method was fixed.	2020-08-09 09:42:05 +03:00
ofen	3fea7c39be	401 Unauthorize HTTP error added (#681 ) 401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]	2020-08-09 09:39:37 +03:00
Aliaksandr Valialkin	d8aa433c4d	vendor: update github.com/VictoriaMetrics/metrics from v1.12.2 to v1.12.3	2020-08-07 13:02:35 +03:00
Aliaksandr Valialkin	67cacb22ac	lib/httpserver: add `-tls`, `-tlsCertFile` and `-tlsKeyFile` command-line flags in every vm binary This makes such binaries compatible with binaries from `master` branch (aka single-node version) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677	2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin	307281e922	lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit This should improve data ingestion performance when heavy searches are executed See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin	dd1d59f57a	lib/storage: properly check timeouts and pace limits Previously they were checked on every iteration for small number of iterations	2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin	2704722b6d	deployment/docker: update Go builder from v1.14.6 to v1.14.7	2020-08-07 08:29:25 +03:00
Aliaksandr Valialkin	87d7710c8e	docs/MetricsQL.md: mention that MetricsQL removes all the NaN values from results	2020-08-07 07:51:40 +03:00
Aliaksandr Valialkin	95a8c492ef	app/vmselect/promql: properly handle `-n^m` like Prometheus does `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin	7f93d61a56	app/vmselect/promql: remove metric name after applying `clamp_min` and `clamp_max` functions in order to be consistent with Prometheus This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin	01000505a0	app/vmselect/promql: remove metric name after applying `ceil`, `floor` and `round` functions in order to be more consistent with Prometheus This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin	75bff1a567	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin	8835004a4c	app/vmselect: properly handle PromQL queries like `scalar1 < metric < scalar2` like Prometheus does This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin	3947307363	vendor: update github.com/VictoriaMetrics/metricsql from v0.2.10 to v0.3.0 This adds support for special integers in MetricsQL that start from 0x, 0b, 0o. This improves compatibility with PromQL - see https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 21:45:33 +03:00
Aliaksandr Valialkin	a2039b3bbc	app/vmselect: return the upper bound on the number of found time series from storage.Search.Init This is used by a single-node version in order to reduce memory allocations during search. See `bc8381613d` for details.	2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin	b690eeff53	lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x	2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin	6c0a92a1ee	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin	14ddb8a34e	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin	46c98cd97a	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin	13f8644f8e	lib/storage: optimize prefetching metric names for the given metricIDs	2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin	a455930ab4	app/vmstorage: rename `vm_cache_size_entries{type="storage/prefetchedMetricIDs"}` to `vm_cache_entries{type="storage/prefetchedMetricIDs"}` to be consistent with other `vm_cache_entries` metrics	2020-08-06 16:34:18 +03:00
Aliaksandr Valialkin	f789e0fa44	lib/fs: export `vm_nfs_pending_dirs_to_remove` metric for monitoring the number of pending directories that couldn't be removed due to NFS lock	2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin	a3e91c593b	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin	76064ba9e7	Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer	2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin	9038df211f	vendor: `make vendor-update`	2020-08-05 11:15:47 +03:00
Aliaksandr Valialkin	a930460236	app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663	2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin	a04f4a3d9a	app/vmselect: use warning level instead of info level for logging slow queries that take longer than `-search.logSlowQueryDuration`	2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin	265206a6c3	docs/Single-server-VictoriaMetrics.md: add a chapter about data updates	2020-08-04 13:54:29 +03:00
Aliaksandr Valialkin	8cc2e01386	lib/backup: allow using `~/.aws/config` without region Use us-west-2 for determining bucket region.	2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin	bdb881c43b	app/vmselect/promql: add zscore-related functions: `zscore_over_time(m[d])` and `zscore(q) by (...)`	2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin	94471a1273	app: remove duplicate *-pure makefile rules	2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin	a2aa3a60eb	app/vmselect: show `X-Forwarded-For` contents on `/api/v1/status/active_queries` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin	3149af624d	lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this increases query latencies since more CPUs are busy with merges. While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.	2020-07-31 17:53:13 +03:00
Aliaksandr Valialkin	106e302d7a	all: add mssing APP_NAME to vm*-GOARCH builds	2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin	945645f38f	docs/{vmagent,vmalert}: add instruction on how to build for ARM	2020-07-31 09:25:41 +03:00
Aliaksandr Valialkin	a0eec52e6c	docs/Single-server-VictoriaMetrics.md: mention that downgrade is also safe to perform	2020-07-31 09:21:03 +03:00
Aliaksandr Valialkin	bbc7583015	vendor: update github.com/valyala/quicktemplate from v1.5.2 to v1.6.0	2020-07-30 23:39:48 +03:00
Aliaksandr Valialkin	0c00fe70cf	app/vmselect: do not adjust `start` and `end` query args passed to `/api/v1/query_range` when `-search.disableCache` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin	e9860b2fa3	docs/vmalert.md: sync with app/vmalert/README.md	2020-07-30 21:57:20 +03:00
Aliaksandr Valialkin	29bbab0ec9	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin	96039dcb40	lib/storage: properly update `vm_slow_row_inserts_total` metric when importing multiple data points per time series at once Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series, while only a single increment is needed when inserting the first data point for this time series.	2020-07-30 16:17:19 +03:00
Aliaksandr Valialkin	bafd475f2c	vendor: update github.com/valyala/quicktemplate from v1.5.1 to v1.5.2	2020-07-29 18:20:19 +03:00
Aliaksandr Valialkin	1e067401ba	lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-29 13:12:35 +03:00
Aliaksandr Valialkin	338ee47d60	app/vmselect/promql: return non-empty value from `rate_over_sum(m[d])` even if a single data point is located in the given `[d]` window Just divide the data point value by the window duration in this case.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	717c554fb0	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in `increase()` is handled in another way now See `e00cfc854d` for the approach used now.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	d9037b3970	app/vmselect/promql: fill gaps with 0 in `rate_over_sum` response when the last value before the selected time window isnt empty	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	f068ea74d0	vendor: `make vendor-update`	2020-07-29 09:35:34 +03:00
Aliaksandr Valialkin	ab47fa2300	vendor: update github.com/VictoriaMetrics/metrics from v1.12.1 to v1.12.2	2020-07-28 22:01:41 +03:00
Aliaksandr Valialkin	f6d4275087	app/{vmagent,vminsert}: properly preserve `db` tag from query string passed to Influx line protocol query Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:25:49 +03:00
Aliaksandr Valialkin	baebe86844	app/vmagent/remotewrite: add missing `resp.Body.Close()` after pushing data to remote storage Missing body close could disable HTTP keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin	0f6f0d30d3	app/vmselect: show query origin (aka remote_addr or client address) on the `/api/v1/status/active_queries` page for every query	2020-07-28 15:14:40 +03:00
Roman Khavronenko	ec6ed467c6	app/vmalert: support `external.label` to specify global labelset for all rules #622 (#652 ) `external.label` flag supposed to help to distinguish alert or recording rules source in situations when more than one `vmalert` runs for the same datasource or AlertManager.	2020-07-28 14:23:04 +03:00
Aliaksandr Valialkin	9dccedc599	app/vmselect/promql: return empty values from `group()` if all the time series have no values at the given timestamp This aligns `group()` behaviour to Prometheus	2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin	312acf7ce9	docs/MetricsQL.md: small fixes in the docs	2020-07-28 13:28:01 +03:00
Aliaksandr Valialkin	91d4673bd6	docs/Single-server-VictoriaMetrics.md: mention that OpenTSDB data ingestion protocol is used by KairosDB	2020-07-28 13:11:36 +03:00
Aliaksandr Valialkin	ce7c8898af	vendor: update github.com/VictoriaMetrics/metrics from v1.12.0 to v1.12.1	2020-07-28 00:21:06 +03:00
Sasasu	96bc476e53	lib/storage: metaindexRow use memroy more efficiently (#655 ) due to memory align the metaindexRow structure use 64-byte pre object. this commit changes the order of field, make metaindexRow use 56-byte pre object. Signed-off-by: Sasasu <su@sasasu.me>	2020-07-27 23:23:25 +03:00
Aliaksandr Valialkin	f26ef58137	lib/protoparser/prometheus: add a test for cassandra-exporter Thanks to Seva	2020-07-27 18:37:46 +03:00
Aliaksandr Valialkin	d5057f6d04	app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage Previously the request body was already consumed before the retry, so this led to the following error: http: ContentLength=... with Body length 0	2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin	b191e425b3	app/vmselect/promql: improve further the accuracy of buckets_limit() function The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin	43871e79c6	app/vmselect/promql: avoid dropping `inf` bucket in `buckets_limit` The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin	978c1e930e	app/vmselect/promql: optimize `buckets_limit(k, buckets)` for big number of buckets	2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin	cc735da814	deployment/docker/docker-compose.yml: update Grafana version from 7.0.3 to 7.1.1	2020-07-24 18:41:49 +03:00
Aliaksandr Valialkin	51cbf27077	app/vmselect/promql: improve the accuracy of `buckets_limit(k, buckets)` function Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin	cf69b1ea6f	app/vmselect/promql: add `buckets_limit(k, buckets)` function, which limits the number of buckets per time series to `k` This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin	45334f61de	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin	3526e8768a	app/vmselect/promql: typo fix after `3e557c9861`	2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin	94cc677b0c	lib/storage: slightly reduce code difference between single-node and cluster versions	2020-07-24 01:18:05 +03:00
Aliaksandr Valialkin	8d1721d128	app/vmselect/promql: add `rate_over_sum(m[d])` function to MetricsQL, which returns rate over sum of `m` values over `d` duration Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin	88e8bed0c9	app/vmselect/promql: allow setting `[d]` window smaller than the interval between raw points for `avg_over_time` This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin	fb3d1380ac	lib/storage: respect `-search.maxQueryDuration` when searching for time series in inverted index Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin	dbf3038637	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin	16a4b1b20c	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks	2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin	0750d2cec1	app/vminsert: export `vm_relabel_metrics_dropped_total` metric that shows the number of metrics dropped due to relabeling	2020-07-23 14:58:02 +03:00
Aliaksandr Valialkin	55ed07add1	app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g	2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin	7aa5b48508	app/vmselect: reduce memory usage when querying big number of time series with long labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin	49a0011837	app/vminsert: do not call ApplyRelabeling function if relabeling is disabled This should reduce CPU usage a bit when `-relabelConfig` isn't set	2020-07-23 13:35:36 +03:00
Aliaksandr Valialkin	c91ccce50c	app/vminsert: fix relabeling for metrics ingested via Influx line protocol Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels if a single Influx line protocol message contains multiple field values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638	2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin	b8303afcd8	lib/storage: improve prioritizing of data ingestion over querying Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin	7d0743422b	lib/storage: properly calculate global metrics in UpdateStats()	2020-07-23 00:35:31 +03:00
Aliaksandr Valialkin	6afdcf8a20	lib/mergeset: properly calculate global metrics in UpdateStats() Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.	2020-07-23 00:35:29 +03:00
Aliaksandr Valialkin	23fa44e56e	lib/storage: reorder mergeBlockStreams() args in order to make them more consistent	2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin	754eac676d	lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs This condition may occur after the following sequence of events: 1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs. 2) All the goroutines return from Storage.AddRows. 3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body. The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal(). This may take indefinite time.	2020-07-22 21:52:42 +03:00
Aliaksandr Valialkin	71c3266fca	docs/Single-server-VictoriaMetrics.md: mention that it is recommended inspecting logs during troubleshooting	2020-07-22 18:21:06 +03:00
Aliaksandr Valialkin	edbc777e91	vendor: `make vendor-update`	2020-07-22 16:54:02 +03:00
Aliaksandr Valialkin	20d0c41ac5	app/vmselect/prometheus: support `d`, `w` and `y` suffixes for durations passed to `step` in `/api/v1/query_range` like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin	bd4299fafe	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile	2020-07-22 15:04:42 +03:00
Roman Khavronenko	9460bf782e	dashboards/victoriametrics: update `Concurrent inserts` panel #632 (#645 ) Panel `Concurrent inserts` was moved to `vminsert` row. Its metrics and description was updated accordingly to #632 issue.	2020-07-22 12:43:23 +03:00
Aliaksandr Valialkin	a3f48e395e	app/vmagent: add `-remoteWrite.decimalPlaces` command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin	67be79a0bc	lib/uint64set: optimize adding items to the set via Set.AddMulti	2020-07-21 20:57:05 +03:00
Aliaksandr Valialkin	5bb4fe1ba4	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin	0755cb3b50	app/vmselect/promql: skip the first value in time series passed to `increase()` if it exceeds by more than 10x the delta between the next value and the first value This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin	71eba8dcf5	app/vmselect: log the total available memory for concurrent requests on `not enough memory` errors This should simplify root cause analysis	2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin	3b246aa569	app/vmagent: add `-remoteWrite.proxyURL` command-line option This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy. This is similar to `proxy_url` option in `remote_write` section of Prometheus. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write	2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin	8bee3ef91b	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-20 17:09:30 +03:00
Roman Khavronenko	8949ec961d	app/vmagent: mention grafana dashboard in README (#639 )	2020-07-20 17:09:27 +03:00
Aliaksandr Valialkin	77523af9fc	vendor: update github.com/VictoriaMetrics/metrics from v1.11.3 to v1.12.0	2020-07-20 16:56:44 +03:00
Aliaksandr Valialkin	86b54f3768	app/vmagent/remotewrite: allow passing empty `-remoteWrite.urlRelabelConfig` entries	2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin	141e84b5a4	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin	4d2011a87d	app/vmselect/promql: add `mode()` aggregate function	2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin	31ef39e8da	lib/httpserver: log remote address in error message from `httpserver.Errorf` This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin	427fa43ce2	app/vmselect/promql: add `mode_over_time(m[d])` function See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin	eb402a17bd	app/vmselect/promql: optimize `group(rollup(m))` calculations	2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin	ea8dc85ba8	app/vmselect/promql: check that `any()` doesn't touch metric name	2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin	b8b13e82e0	deployment/docker: update Go builder from v1.14.5 to v1.14.6 This fixes runtime issues found in Go since v1.14.5. See https://github.com/golang/go/issues?q=milestone%3AGo1.14.6+label%3ACherryPickApproved	2020-07-17 15:21:12 +03:00
Aliaksandr Valialkin	fc8fe38a82	app/vmselect/promql: add `group()` aggregate function to MetricsQL This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin	c64914a7e4	app/vmselect/promql: keep all labels for time series from `any()` call	2020-07-17 15:17:37 +03:00
Roman Khavronenko	21cf6a1ec4	deployment/docker: replace Prometheus with vmagent (#635 ) * replace Prometheus with vmagent in docker compose env; * cluster dashboard: exclude vmagent from job list; * cluster dasbhoard: reference datasource var instead of datasource name.	2020-07-17 02:18:03 +03:00
Roman Khavronenko	87946dcc53	vmagent: update grafana dashboard (#634 ) * reference datasource variable instead of datasource name; * change unit from `bytes` to `bits/s` for Network panel.	2020-07-17 02:12:20 +03:00
Aliaksandr Valialkin	f9b38f7f2d	app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call Previously this could lead to `out of range` panic	2020-07-17 00:05:24 +03:00
Aliaksandr Valialkin	14dc426b45	app/vmselect: fix `nil pointer dereference` panic when unsuccessfully querying `vmstorage`	2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin	490a42f592	deployment/docker: update Go builder from v1.14.4 to v1.14.5 This should fix the following issues in Go - https://github.com/golang/go/issues?q=milestone%3AGo1.14.5+label%3ACherryPickApproved	2020-07-16 18:55:37 +03:00
Roman Khavronenko	cb4c433260	vmagent: add grafana dashboard (#629 ) `vmagent` Grafana dashboard suppose to provide basic observability over multiple `vmagent` instances. Dashboard is saved in Grafana export format so it can be easily imported. It was also integrated into docker-compose environment.	2020-07-15 13:58:30 +03:00
Aliaksandr Valialkin	ce381b3868	app/vmalert: consistently use "%w" instead of "%s" in `fmt.Errorf` when wrapping errors	2020-07-15 13:55:13 +03:00
Aliaksandr Valialkin	e6d96bb0bd	docs/vmagent.md: make filtering rules for init container pods less confusing	2020-07-14 20:33:19 +03:00
Aliaksandr Valialkin	74fb0b293d	vendor: `make vendor-update`	2020-07-14 20:30:57 +03:00
Aliaksandr Valialkin	8e7c7a6fbd	docs/Single-server-VictoriaMetrics.md: remove `Roadmap` chapter, since it became outdated	2020-07-14 19:07:06 +03:00
Aliaksandr Valialkin	c2b4b9138d	app/vmagent/remotewrite: return proper value from `tssRelabelPool.New` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin	3365f6867b	docs/Single-server-VictoriaMetrics.md: sync with README.md	2020-07-14 14:19:53 +03:00
Aliaksandr Valialkin	86044f6561	app/{vminsert,vmagent}: add `-influxSkipMeasurement` command-line flag for using field name as metric name See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626	2020-07-14 14:18:40 +03:00
Aliaksandr Valialkin	be0ab4fbfe	lib/storage: reset `MetricName->TSID` cache after marking metricIDs as deleted This is a follow-up commit after `12b16077c4` , which didn't reset the `tsidCache` in all the required places. This could result in indefinite errors like: missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time Fix this by resetting the cache inside deleteMetricIDs function.	2020-07-14 14:05:19 +03:00
Aliaksandr Valialkin	0e7b2008b2	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin	a4c96d9e6d	lib/protoparser: properly update `vm_protoparser_rows_read_total{type="promscrape"}` metric	2020-07-14 12:15:56 +03:00
Seva Poliakov	a5e713b6e0	add vm_protoparser_rows_read_total metrics to promscrape (#624 ) * add vm_protoparser_rows_read_total metrics to promscrape move vm_protoparser_rows_read_total for promscrape to better place move vm_protoparser_rows_read_total for promscrape to better place * remove possibility of infinity loop at prometheus parser	2020-07-14 12:02:25 +03:00
Roman Khavronenko	207e93b50d	lib/flagutil: specify additional description for all Array type flags (#620 ) Array type flag is now defined as `value` type in flag description when printed. This change adds additional description to every Array type flag so it would be clear what exact type is used: ``` -remoteWrite.urlRelabelConfig array Optional path to relabel config for the corresponding -remoteWrite.url Supports array of values separated by comma or specified via multiple flags. ```	2020-07-13 22:00:03 +03:00
Roman Khavronenko	605711bde5	lib/persistentqueue: add `vm_persistentqueue_bytes_pending` metric (#619 ) Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount of bytes in persistentqueue flushed on disk as a difference between write and read offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes` except of accounting for bytes in-memory.	2020-07-13 21:54:54 +03:00
Roman Khavronenko	a02097e657	Extend metric `vm_promscrape_targets` with `status` label (#615 ) The change to `vm_promscrape_targets` metric suppose to improve observability for `vmagent` so it will be possible to track how many targets are up or down for every specific scrape group: ``` vm_promscrape_targets{type="static_configs", status="down"} 1 vm_promscrape_targets{type="static_configs", status="up"} 2 ```	2020-07-13 21:54:53 +03:00
Aliaksandr Valialkin	3898cc0285	app/vmselect/prometheus: minimize the diff for the change `1033dc7e2a` over `619b0a25c9`	2020-07-13 21:41:17 +03:00
faceair	bf39e67ade	fix empty response template (#617 )	2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin	b6a5c29549	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-13 21:26:00 +03:00
ofen	9ffa688846	Update README.md (#621 ) Troubleshooting section updated to help out with duplicate targets detection	2020-07-13 21:25:59 +03:00
Aliaksandr Valialkin	4353ff7ef1	app/vmagent: fix data race when multiple `-remoteWrite.urlRelabelConfig` options are set Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race and improper relabeling. Now each goroutine has its own copy of tss during relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-10 15:17:23 +03:00
Aliaksandr Valialkin	805a90f642	app/vmagent/remotewrite: typo fix in `-remoteWrite.showURL` help message	2020-07-10 14:07:14 +03:00
Aliaksandr Valialkin	5910207d61	vendor: update github.com/valyala/quicktemplate from v1.5.0 to v1.5.1 This should fix incorrect encoding for json strings with char codes below 0x20 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/613	2020-07-10 12:58:40 +03:00
Aliaksandr Valialkin	5d21c79af9	docs/Single-server-VictoriaMetrics.md: sync with the original README.md	2020-07-10 12:16:16 +03:00
Aliaksandr Valialkin	6373d377ef	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via `/api/v1/import/prometheus`	2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin	2012e294d1	properly calculate readCalls	2020-07-10 12:01:05 +03:00
Aliaksandr Valialkin	d449d0a0e1	app/vmselect/promql: add missing tests for `ifnot` binary operation	2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin	7e706eea13	app/vmselect/promql: refactor implementations for `and` and `unless` binary operations, so they are closer to `or` implementation	2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin	6c1a47b5e0	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin	418f0e46cb	docs: add a link to the `The CMS monitoring infrastructure and applications` publication from CERN	2020-07-08 20:16:31 +03:00
Aliaksandr Valialkin	87f8c728bf	lib/promscrape: send `Accept` header similar to Prometheus when scraping targets This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:50:06 +03:00
Aliaksandr Valialkin	fd4d593c75	vendor: `make vendor-update`	2020-07-08 19:24:59 +03:00
Aliaksandr Valialkin	cd58e4356d	docs/Cluster-VictoriaMetrics.md: mention about `api/v1/status/active_queries` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528	2020-07-08 19:15:38 +03:00
Aliaksandr Valialkin	fb86071552	app/vmselect: add `/api/v1/status/active_queries` page with the list of currently running queries This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528	2020-07-08 19:09:31 +03:00
DexterZhang	9930ce1fa9	Feat/query list vmselect (#575 ) * feat(vmselect): add support for listing current running queries and canceling specific query * fix(vmselect): change current queries' pid from int64 counter to uuid * feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth * fix(vmselect): add more info to current queries * review: delete some unnecessary code and use function instead of init * review: returen queriesMap in newQueriesMap review: delete unused var in struct queriesMap, add comments to exported functions * review: add return if error occurs * feat(vmselect): truncate query string in current running query list API since the size of query string might be large; use query string's pointer in struct `query` for the same reason; add query info API to get full access of query's info;	2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin	7335743d57	lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense, since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs, i.e. to GOMAXPROCS.	2020-07-08 17:34:27 +03:00
Roman Khavronenko	929ad74de6	lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607 ) The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to respect the naming standard.	2020-07-08 14:19:27 +03:00
Aliaksandr Valialkin	e401b8d527	lib/protoparser/graphite: go fmt	2020-07-08 14:13:06 +03:00
Aliaksandr Valialkin	50ecf09042	lib/protoparser/graphite: add more tests after `eb45185eef` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610	2020-07-08 14:13:03 +03:00
Seva Poliakov	1ae0334e17	Fix graphite minus one timestamp (#609 ) * fix graphite -1 timestamp * format the graphite fix -1 timestamp	2020-07-08 14:13:01 +03:00
Aliaksandr Valialkin	fad008df7e	lib/storage: clarify `out of retention period` error message by mentioning `-retentionPeriod` command-line flag	2020-07-08 13:54:13 +03:00
Aliaksandr Valialkin	fe58462bef	lib/storage: reset MetricName->TSID cache after deleting time series This should prevent from adding new data points to deleted time series without the need to check for the deleted time series. This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set contains big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596 Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604	2020-07-06 22:01:24 +03:00
Aliaksandr Valialkin	77bb0e6595	lib/fs: clarify description for `-fs.disableMmap` command-line flag	2020-07-06 14:28:57 +03:00
Aliaksandr Valialkin	0bff96fe4b	lib/storage: prioritize data ingestion over heavy queries Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:44:04 +03:00
Roman Khavronenko	9afd19d375	app/vmalert: add retries to remotewrite (#605 ) * app/vmalert: add retries to remotewrite Remotewrite pkg now does limited number of retries if write request failed. This suppose to make vmalert state persisting more reliable. New metrics were added to remotewrite in order to track rows/bytes sent/dropped. defaultFlushInterval was increased from 1s to 5s for sanity reasons. * fix * wip * wip * wip * fix bits alignment bug for 32-bit systems * fix mistakenly dropped field	2020-07-05 18:47:38 +03:00
Aliaksandr Valialkin	82871fb7a5	app/vmselect/prometheus: small fixes on top of `8bb762124a`	2020-07-05 18:17:53 +03:00
faceair	17f175ff5a	fix adjust last points avoid influence earlier value (#606 )	2020-07-05 18:17:52 +03:00
Aliaksandr Valialkin	6f1d926698	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:48 +03:00
Aliaksandr Valialkin	ee03b4ccbd	lib/promscrape: prevent from too big deadline misses on scrape retries The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:42:09 +03:00
Aliaksandr Valialkin	dfa83a4a35	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:25 +03:00
Ween	d28fb0baf9	[VMAlert] Fix error log when remoteWrite queue size is full (#602 ) * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors * fix error logs when queue is full Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-07-03 16:50:43 +03:00
Aliaksandr Valialkin	8bb3622e9d	app/vminsert: prevent from adding and/or selecting labels with empty values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin	6ebac3ab63	app/vminsert: add ability to apply relabeling to all the incoming metrics if `-relabelConfig` command-line arg points to a file with a list of `relabel_config` entries See https://victoriametrics.github.io/#relabeling	2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin	a45856570b	all: typo fix: exptected -> expected	2020-07-02 18:06:21 +03:00
Aliaksandr Valialkin	f10e8809c0	app/vmselect: add `interpolate` function for filling gaps with linearly interpolated values See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin	2361ad8ab4	lib/promscrape: add ability to set `disable_compression` and `disable_keepalive` options in `scrape_config` section of the config passed to `-promscrape.config` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:34 +03:00
Aliaksandr Valialkin	0f754bea49	lib/promscrape: add `-promscrape.disableKeepAlive` command-line flag for disabling http keep-alive connections when scraping targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-01 02:20:46 +03:00
BigFish	aa26b94f33	fix: spelling mistakes (#594 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-07-01 01:36:40 +03:00
Aliaksandr Valialkin	618bcc818c	vendor: `make vendor-update`	2020-07-01 01:04:05 +03:00
Aliaksandr Valialkin	4cb3e7595c	app/vmstorage: add `-denyQueriesOutsideRetention` command-line flag for denying queries outside the configured retention	2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin	81e3d4305f	lib/httpserver: add Unwrap method to ErrorWithStatusCode, so `As` and `Is` functions in standard `errors` package may properly unwrap the error inside ErrorWithStatusCode	2020-07-01 00:53:49 +03:00
Aliaksandr Valialkin	fe77d661b3	all: use `errors.As` instead of type assertion for detecting `net.Error`	2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin	0c4e8aeb2b	all: use `errors.As` for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin	d962568e93	all: use %w instead of %s for wrapping errors in `fmt.Errorf` This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin	5a43842bd3	lib/promscrape: add missing label sorting for autogenerated metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/592	2020-06-29 22:39:40 +03:00
Ween	b42cf33c4d	Fix Auto metrics relabeled errors (#593 ) * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-06-29 22:39:39 +03:00
Roman Khavronenko	156c83d112	app/vmalert: support multiple notifier urls (#584 ) (#590 ) * app/vmalert: support multiple notifier urls (#584) User now can set multiple notifier URLs in the same fashion as for other vmutils (e.g. vmagent). The same is correct for TLS setting for every configured URL. Alerts sending is done in sequential way for respecting the specified URLs order. * app/vmalert: add basicAuth support for notifier client (#585) The change adds possibility to set basicAuth creds for notifier client in the same fasion as for remote write/read and datasource.	2020-06-29 22:21:56 +03:00
Aliaksandr Valialkin	5341596f96	docs/vmalert.md: sync with app/vmalert/README.md	2020-06-29 22:18:30 +03:00
Roman Khavronenko	bbeab70de6	app/vmalert: move flags description and initialization into subpackages The change adds no new functionality and aims to move flags definitions to subpackages that are using them. This should improve readability of the main function.	2020-06-29 22:18:29 +03:00
kreedom	63c36e2e69	app/vmalert: properly set transport for HTTP clients Fixes issue #586	2020-06-29 22:18:25 +03:00
Aliaksandr Valialkin	2b504f17de	docs: update the info that docker images are built on top of `alpine` image now A follow-up after the commit `ff624c9125` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-26 13:52:25 +03:00
Aliaksandr Valialkin	9b06c83cd6	vendor: `make vendor-update`	2020-06-25 23:44:33 +03:00
Aliaksandr Valialkin	b2b17589fa	vendor: update github.com/valyala/fastjson from v1.5.1 to v1.5.2	2020-06-25 23:34:24 +03:00
Aliaksandr Valialkin	aad38c8283	lib/promrelabel: properly apply `^` and `$` anchors to `regex` value in Prometheus relabeling rules	2020-06-25 17:19:02 +03:00
Aliaksandr Valialkin	a586b8b6d4	app/vminsert/netstorage: do not re-route every time series to more than two vmstorage nodes when certain vmstorage nodes are temporarily slower than the rest of them Previously vminsert may spread data for a single time series across all the available vmstorage nodes when vmstorage nodes couldn't handle the given ingestion rate. This could lead to increased usage of CPU and memory on every vmstorage node, since every vmstorage node had to register all the time series seen in the cluster. Now a time series may spread to maximum two vmstorage nodes under heavy load. Every time series is routed to a single vmstorage node under normal load.	2020-06-25 16:42:37 +03:00
Aliaksandr Valialkin	12b87b2088	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:37:35 +03:00
Aliaksandr Valialkin	d664bde307	deployment/docker/docker-compose.yml: update Prometheus from v1.18.1 to v1.19.1 and Grafana from v7.0.2 to v7.0.3	2020-06-24 18:09:53 +03:00
Aliaksandr Valialkin	2953c0ec76	docs/Cluster-VictoriaMetrics.md: move VictoriaMetrics logo below "Cluster version" heading, since it is heeded for proper navigation at https://victoriametrics.github.io	2020-06-24 12:05:53 +03:00
Aliaksandr Valialkin	8eb2e5384c	docs/SampleSizeCalculations.md: updates	2020-06-24 12:05:52 +03:00
Aliaksandr Valialkin	4931b719d7	docs/SampleSizeCalculations.md: add a doc with calculations for the "Lowest sample size" graph at https://victoriametrics.com/	2020-06-24 12:00:45 +03:00
nicbaz	46c5c0772c	vmselect: fix label_replace when mismatch (#579 ) As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the MetricsQL `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:54:29 +03:00
Aliaksandr Valialkin	fd7a3d880e	lib/fs: go fmt	2020-06-23 23:03:08 +03:00
Aliaksandr Valialkin	08edb90814	lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 22:55:56 +03:00
Aliaksandr Valialkin	1eed50b9ca	docs/vmalert.md: sync with app/vmalert/README.md	2020-06-23 22:48:25 +03:00
nicbaz	ea2ed4b7e8	vmalert: add support for TLS configuration (#578 ) app/vmalert: add support for TLS configuration Add support for TLS optional configuration in a similar fashion to what is currently supported in other vmutils such as vmagent. TLS configuration options are distinct for datasource, remoteRead, remoteWrite as well as notifier.	2020-06-23 22:47:23 +03:00
Aliaksandr Valialkin	0fdbe5de25	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin	3a444bb7bb	lib/promrelabel: add support for `keep_if_equal` and `drop_if_equal` actions to relabel configs These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:19 +03:00
Aliaksandr Valialkin	de7e585ac8	lib/promscrape: preserve the previously discovered targets on discovery errors per each `job_name` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582	2020-06-23 15:42:46 +03:00
Aliaksandr Valialkin	6d9c5ad422	vendor: update github.com/klauspost/compress from v1.10.9 to v1.10.10	2020-06-23 13:47:00 +03:00
Aliaksandr Valialkin	521c657f8d	lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 13:40:20 +03:00
Aliaksandr Valialkin	5fb60dd647	lib/logger: add `-loggerErrorsPerSecondLimit` for limiting the rate of ERROR messages	2020-06-23 12:42:59 +03:00
Aliaksandr Valialkin	a80e852aab	lib/promscrape: retry performing the request to the server for up to 3 times before giving up when it closes keep-alive connections Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-06-23 12:34:12 +03:00
Aliaksandr Valialkin	97557c96d5	docs/Single-server-VictoriaMetrics.md: remove `-httpListenAddr` command-line flag from `setting up VictoriaMetrics` chapter This flag is optional and it has good default value - `:8428`, so there is no need in mentioning it at this chapter	2020-06-22 12:45:50 +03:00
kreedom	f227799c87	Support of custom URL path for alert (#560 ) app/vmalert: Support custom URL for alerts source Add flag `external.alert.source` for configuring custom URL for alert's source. This may be handy to re-point default source URL to other systems like Grafana. Updates #517	2020-06-21 16:33:58 +03:00
Aliaksandr Valialkin	70bf8218bb	app/vmselect/promql: properly override label values from `group_left` and `group_right` lists like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin	50aa34bcbe	lib/promscrape/discovery/consul: reduce load on Consul when discovering big number of targets by using background caching Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 18:20:07 +03:00
Aliaksandr Valialkin	62e1908986	lib/promscrape: reduce default value for `-promscrape.discovery.concurrency` from 500 to 100 This should reduce load on Kubernetes API server and Consul when big number of targets are discovered Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 17:53:48 +03:00
Aliaksandr Valialkin	1f2826bae2	lib/promscrape/discovery/ec2: expose `__meta_ec2_ami` like the next Prometheus release will do See `b5d61fb66c` for details	2020-06-20 17:45:30 +03:00
Aliaksandr Valialkin	2fc2679a3f	app/vminsert/netstorage: remove possible race condition when broken connection may be recovered before acquiring storageNode.bcLock	2020-06-20 16:38:08 +03:00
Aliaksandr Valialkin	85036c2b07	docs/Cluster-VictoriaMetrics.md: add `high availability` chapter	2020-06-20 15:53:07 +03:00
Aliaksandr Valialkin	96e9eed234	docs/Single-server-VictoriaMetrics.md: mention that `vmauth` could be used for routing user requests to particular VictoriaMetrics instances	2020-06-19 16:17:05 +03:00
Aliaksandr Valialkin	2e5212ab95	docs/Single-server-VictoriaMetrics.md: add a link to features available for enterprise customers	2020-06-19 13:18:07 +03:00
Aliaksandr Valialkin	9409a31c07	docs/vmauth.md: mention that we can provide custom integration with SAML	2020-06-19 13:13:53 +03:00
Aliaksandr Valialkin	4400700832	app/vminsert: properly replicate data for the last `RF-1` storage nodes for `-replicationFactor=RF` Previously the data for the last `RF-1` storage noes has been incorrectly replicated to the first storage node.	2020-06-19 12:40:22 +03:00
Aliaksandr Valialkin	ca4c9023e3	vendor: `make vendor-update`	2020-06-19 02:40:36 +03:00
Tristan Su	c254b683fd	lib/storage: set big/small merge concurrency (#568 ) fixed #567 Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>	2020-06-19 02:21:55 +03:00
Aliaksandr Valialkin	2e5b6220a4	lib/promrelabel: allows regex capture groups in target_label like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569	2020-06-19 02:20:58 +03:00
Aliaksandr Valialkin	4f673a5201	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin	af7db914c2	`make docs-sync`	2020-06-18 23:55:04 +03:00
Aliaksandr Valialkin	fd1afa5c63	docs/Articles.md: add a link to article https://stas.starikevich.com/posts/raspberry-pi-4-prometheus/	2020-06-18 23:13:54 +03:00
Aliaksandr Valialkin	6939e36fdd	app/vmselect/promql: fill gaps on right side with values from left side of `or` operator in the same way as Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin	85c1ccb8b8	app/vminsert/netstorage: add missing `return` in storageNode.checkHealth on connection failure	2020-06-18 20:51:51 +03:00
Aliaksandr Valialkin	464682f380	app/vminsert/netstorage: periodically check for each `-storageNode` health, so it could be marked as healthy when it is ready to accept data This fixes uneven data routing in cluster version when `-replicationFactor` is set to 1 (default value), i.e. when the replication is disabled. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-18 20:42:43 +03:00
Aliaksandr Valialkin	5f3a895c23	lib/storage: add `key!=".+"` filter additionally to negative filter matching empty value such as `key!~"\|foo"` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-18 20:05:45 +03:00
Roman Khavronenko	1a01fe2cf2	vmalert-537: allow name duplication for rules within one group. (#559 ) Uniqueness of rule is now defined by combination of its name, expression and labels. The hash of the combination is now used as rule ID and identifies rule within the group. Set of rules from coreos/kube-prometheus was added for testing purposes to verify compatibility. The check also showed that `vmalert` doesn't support `query` template function that was mentioned as limitation in README.	2020-06-18 18:54:35 +03:00
Aliaksandr Valialkin	87151e825e	docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible	2020-06-18 18:54:34 +03:00
Roman Khavronenko	a171f9b03e	dashboard: update cluster-version dashboard. (#558 ) Fix "Bytes per point" panel query #551.	2020-06-12 22:07:28 +03:00
Aliaksandr Valialkin	cc2225cc49	app/vmselect: fix the error after `936f35920a`	2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin	936f35920a	app/vmselect/prometheus: allow returning partial response from `/api/v1/export` if `-search.denyPartialResponse=false` This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.	2020-06-12 21:11:48 +03:00
Aliaksandr Valialkin	35191d8403	docs/vmalert.md: sync with app/vmalert/README.md	2020-06-10 19:37:48 +03:00
Clémence Saussez	0b53e380cf	app/vmalert: fix link to testdata (#547 ) Fix broken link to vmalert test data Signed-off-by: Clemence Saussez <clemence@zen.ly>	2020-06-10 19:37:21 +03:00
Aliaksandr Valialkin	c40f29f783	lib/storage: properly match `{tag!="\|foo"}` filters Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-10 19:34:37 +03:00
Roman Khavronenko	d71b6e6584	vmalert-491: allow to configure concurrent rules execution per group. (#542 ) The feature allows to speed up group rules execution by executing them concurrently. Change also contains README changes to reflect configuration details.	2020-06-09 15:22:11 +03:00
Roman Khavronenko	5c049bf4dd	vmalert-521: allow to disable rules expression validation. (#536 ) This feature may be useful for using `vmalert` with PromQL compatible datasources like Loki.	2020-06-09 15:19:25 +03:00
Aliaksandr Valialkin	60b8ce47ad	vendor: `make vendor-update`	2020-06-06 00:00:40 +03:00
Aliaksandr Valialkin	356845d716	vendor: update github.com/klauspost/compress from v1.10.7 to v1.10.8	2020-06-05 23:52:08 +03:00
Aliaksandr Valialkin	9f55dea162	lib/httpserver: do not flush and do not close gzip writer if response compression is disabled Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 21:37:46 +03:00
Aliaksandr Valialkin	c1be462d42	app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 20:14:07 +03:00
Aliaksandr Valialkin	7680b7155d	app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set	2020-06-05 20:14:05 +03:00
Aliaksandr Valialkin	cf91a94daf	lib/backup: properly create missing parent directories in fs.CreateFile	2020-06-05 19:28:25 +03:00
Aliaksandr Valialkin	ba1f764b29	lib/fs: optimize queries that read recent samples for big number of time series Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data. This improves read performance by up to 4x.	2020-06-05 19:10:22 +03:00
Aliaksandr Valialkin	2358d9e41d	lib/fs: add a benchmark for ReaderAt.MustReadAt	2020-06-05 19:10:21 +03:00
Aliaksandr Valialkin	01719f4949	app/vmstorage/transport: simplify setupTfss in order to prevent the possibility of nil tfs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534	2020-06-05 13:17:26 +03:00
Aliaksandr Valialkin	e4cef1b678	app/vmstorage: prevent from serving conns from vminsert and vmselect after the server is closed Previously it was possible that the connection is served after the server is closed if the following steps are performed: 1) Server accepts new connection. 2) Server.MustClose() is called and successfully finished. 3) Server starts processing the connection accepted at step 1. There could be various crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534 since the storage may be already closed. Now the server closes the connection at step 3 without processing it.	2020-06-05 11:55:48 +03:00
Aliaksandr Valialkin	58069f5a6a	app/vmalert: print brief usage info for `vmalert -help`	2020-06-05 10:43:24 +03:00
Aliaksandr Valialkin	3848ea3a4a	app/vmauth: print brief usage info for `vmauth -help`	2020-06-05 10:40:11 +03:00
Aliaksandr Valialkin	8ad8ca350a	app/vmagent: print brief usage info for `vmagent -help`	2020-06-05 10:40:10 +03:00
Aliaksandr Valialkin	8b0d9df51d	lib/backup/fsremote: create all the parent directories before creating file in CreateFile	2020-06-05 10:25:28 +03:00
Aliaksandr Valialkin	a1d841b33e	docs/Cluster-VictoriaMetrics.md: remove obsolete line	2020-06-04 20:20:33 +03:00
Aliaksandr Valialkin	4ef0b1181c	docs/Cluster-VictoriaMetrics.md: update stale info about replication	2020-06-04 20:19:12 +03:00
Aliaksandr Valialkin	d49f0597f5	deployment/docker: update Go builder from v1.14.3 to v1.14.4 This fixes the following issue in Go runtime, which could result in program hang - https://github.com/golang/go/issues/38931	2020-06-04 18:07:43 +03:00
Aliaksandr Valialkin	70fe337e7f	docs/Cluster-VictoriaMetrics.md: clarify simultaneous usage of replication and deduplication	2020-06-04 18:00:42 +03:00
Aliaksandr Valialkin	3d0a0b3785	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:14:04 +03:00
DexterZhang	fa103875a0	feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total` (#527 ) * feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total` * refactor(vmselect): use free space instead of used space in tmp block file metrics * fix: add `bytes` suffix to tmp dir free space metric	2020-06-04 13:05:50 +03:00
Vyacheslav Mitrofanov	89a922fb19	allow to use values lower than 10 with the flag -memory.allowedPercent (#531 ) Co-authored-by: Vyacheslav Mitrofanov <vmitrofanov@mfms.ru>	2020-06-03 23:40:13 +03:00
Denis	21df9025c9	Update docker-compose.yml (#530 ) Update to latest version of Prometheus & Grafana.	2020-06-03 23:38:11 +03:00
Aliaksandr Valialkin	faea804b88	app/vmauth: log when -auth.config is reloaded in SIGHUP	2020-06-03 23:22:20 +03:00
Aliaksandr Valialkin	730e4a719f	docs/Single-server-VictoriaMetrics.md: clarify `Replication` section	2020-06-03 21:33:02 +03:00
Aliaksandr Valialkin	e9b9aa4db4	docs/FAQ.md: add a question about the difference between `vmagent` and `Prometheus`	2020-06-03 20:56:59 +03:00
Aliaksandr Valialkin	6637641dd8	docs/Cluster-VictoriaMetrics.md: update `Replication and data safety` chapter	2020-06-03 20:24:21 +03:00
Aliaksandr Valialkin	79adb2dbc7	docs/Single-server-VictoriaMetrics.md: mention vmagent in `high availability` section	2020-06-03 20:16:36 +03:00
Aliaksandr Valialkin	304f9499cf	lib/bytesutil: prevent from garbage collecting s before returning from ToUnsafeBytes	2020-06-03 00:23:27 +03:00
Aliaksandr Valialkin	91cebdccde	vendor: update github.com/klauspost/compress from v1.10.6 to v1.10.7	2020-06-03 00:11:25 +03:00
Aliaksandr Valialkin	1aa0eefd18	docs/Single-server-VictoriaMetrics.md: sync with master	2020-06-02 22:47:04 +03:00
Aliaksandr Valialkin	2961e71217	deployment/docker: use alpine base image for docker images in order to improve debuggability Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-02 22:41:17 +03:00
Aliaksandr Valialkin	32f930d5e7	docs: update FAQ.md	2020-06-02 19:59:18 +03:00
Aliaksandr Valialkin	560ae3c82b	docs/vmalert.md: sync with app/vmalert/README.md via `make docs-sync`	2020-06-02 19:12:53 +03:00
Aliaksandr Valialkin	2ad84be7a3	Makefile: add `make docs-sync` command for syncing docs contents	2020-06-02 19:12:49 +03:00
Aliaksandr Valialkin	045b87c662	app/vmalert: fix comment for UpdateWith exported methods	2020-06-01 14:35:03 +03:00
Aliaksandr Valialkin	43b14b9569	app/vminsert/netstorage: free up unused memory in buffer after memory usage spikes	2020-06-01 14:33:35 +03:00
Roman Khavronenko	44c51c627f	vmalert: Add recording rules support. (#519 ) * vmalert: Add recording rules support. Recording rules support required additional service refactoring since it wasn't planned to support them from the very beginning. The list of changes is following: * new entity RecordingRule was added for writing results of MetricsQL expressions into remote storage; * interface Rule now unites both recording and alerting rules; * configuration parser was moved to separate package and now performs more strict validation; * new endpoint for listing all groups and rules in json format was added; * evaluation interval may be set to every particular group; * vmalert: uncomment tests * vmalert: rm outdated TODO * vmalert: fix typos in README	2020-06-01 13:53:46 +03:00
Aliaksandr Valialkin	37aa4fe282	app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on `/-/reload` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518	2020-05-30 14:37:02 +03:00
Aliaksandr Valialkin	a646131a33	app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent	2020-05-30 14:22:38 +03:00
Aliaksandr Valialkin	f41a01332a	app/vminsert/netstorage: evenly distribute rerouted rows among all the availalbe storage nodes Previously such rows were distributed to the original storage node or to the next storage node. This may result to uneven load among the remaining storage nodes.	2020-05-30 13:51:09 +03:00
Aliaksandr Valialkin	02b2064d8e	app/vminsert/netstorage: do not increment vm_rpc_rows_lost_total when all the vmstorage nodes are unavailable, since vminsert retries sending the data instead of dropping it	2020-05-28 22:36:56 +03:00
Aliaksandr Valialkin	6f94fb6842	docs/CaseStudies.md: add a link to articles about VictoriaMetrics	2020-05-28 21:32:58 +03:00
Aliaksandr Valialkin	c6047b6aa0	docs/Articles.md: added a link to https://medium.com/@IG1.com/sismology-iguana-solutions-monitoring-system-f46e4170447f	2020-05-28 20:10:12 +03:00
Aliaksandr Valialkin	7a61357b5d	app/vminsert/netstorage: make sure that the the data is always replicated among -replicationFactor vmstorage nodes Previously vminsert could write multiple copies of the data to a single vmstorage node when the ingestion rate exceeds the maximum throughput for connections to vmstorage nodes.	2020-05-28 19:59:07 +03:00
Aliaksandr Valialkin	981caa6f0b	docs/Cluster-VictoriaMetrics.md: mention that `opentsdb/api/put` handler is disabled by default	2020-05-28 14:27:17 +03:00
Aliaksandr Valialkin	eca1afdc20	lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0	2020-05-28 11:58:47 +03:00
Aliaksandr Valialkin	b0131c79b6	lib/storage: improve search speed for time series matching Graphite whildcards such as `foo..bar.baz` Add index for reverse Graphite-like metric names with dots. Use this index during search for filters like `__name__=~"foo\\.[^.]\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case. This change may "hide" historical time series during queries. The workaround is to add `[.]` to the end of regexp label filter, i.e. "foo\\.[^.]\\.bar\\.baz" should be substituted with "foo\\.[^.]\\.bar\\.baz[.]".	2020-05-27 21:48:08 +03:00
Aliaksandr Valialkin	fc32881105	vendor: `make vendor-update`	2020-05-27 18:41:21 +03:00
Aliaksandr Valialkin	b09b5f671e	docs/Cluster-VictoriaMetrics.md: mention that `nginx` can be used as a load balancer in front of `vminsert` and `vmselect`	2020-05-27 18:09:39 +03:00
Aliaksandr Valialkin	7bb00cd988	docs: refresh docs about replication support	2020-05-27 17:48:37 +03:00
Aliaksandr Valialkin	77e5165e7b	app/vminsert: add `-replicationFactor` command-line flag for enabling data replication among available -storageNode instances	2020-05-27 17:29:44 +03:00
Aliaksandr Valialkin	b4e3bffe4b	app/vminsert/netstorage: emit warnings instead of errors when re-routing data to healthy storage nodes	2020-05-27 16:31:41 +03:00
Aliaksandr Valialkin	75f2f3b09d	app/vminsert/netstorage: improve ingestion performance when a single vmstorage node is slower than other vmstorage nodes Previously the ingestion performance has been limited by the slowest vmstorage node. Now vminsert should re-route data from the slowest vmstorage node to the remaining nodes.	2020-05-27 15:08:22 +03:00
Aliaksandr Valialkin	9844845d79	app/vminsert: tune the maximum summary buffer size for pending data to 1/4 of available RAM, since 1/2 of RAM is too big considering GOGC overhead	2020-05-25 02:00:37 +03:00
Aliaksandr Valialkin	4a82631e44	app/vminsert: limit the summary buffer sizes for all the storage nodes to a half of the allowed memory	2020-05-25 01:39:33 +03:00
Aliaksandr Valialkin	97feac596f	vendor: `make vendor-update`	2020-05-25 00:06:22 +03:00
Aliaksandr Valialkin	301838e7b1	lib/httpserver: properly set status code for empty response	2020-05-24 23:55:55 +03:00
Aliaksandr Valialkin	64bec11c91	lib/httpserver: fix compression for static files	2020-05-24 22:16:51 +03:00
Aliaksandr Valialkin	99b634e0f9	docs/Single-server-VictoriaMetrics.md: add a video to Zerodha talk about monitoring k8s with VictoriaMetrics	2020-05-24 15:52:13 +03:00
Aliaksandr Valialkin	b747362936	lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response	2020-05-24 14:41:24 +03:00
Aliaksandr Valialkin	fbdce0c6ac	docs/Cluster-VictoriaMetrics.md: mention that cluster components may be monitored with vmagent	2020-05-23 14:29:27 +03:00
Aliaksandr Valialkin	319feb4796	docs/CaseStudies.md: add a link to a post about VictoriaMetrics histograms in Zerodha case study	2020-05-23 12:44:32 +03:00
Aliaksandr Valialkin	cc05d0a3b1	docs/CaseStudies.md: add Zerodha case based on `monitoring K8s with VictoriaMetrics` slides at https://docs.google.com/presentation/d/1g7yUyVEaAp4tPuRy-MZbPXKqJ1z78_5VKuV841aQfsg/edit	2020-05-23 12:41:54 +03:00
Aliaksandr Valialkin	4bd3d4b148	app/vminsert/netstorage: do not return error from storageNode.flushBufLocked when the buffer has been successfully re-routed to healthy nodes This should reduce the number of false errors in the log and the number of falsely lost rows	2020-05-22 18:29:43 +03:00
Aliaksandr Valialkin	6edc33d9bb	app/vminsert/netstorage: capture the first error instead of the last error when sending data to vmstorage The first error has more chances to point to the real root cause of the issue.	2020-05-22 17:49:33 +03:00
Aliaksandr Valialkin	be7253c084	lib/httpserver: do not recompress already compressed response This shoud help with vmauth issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/514	2020-05-22 16:45:20 +03:00
Aliaksandr Valialkin	bb4a2bf1aa	app/vmauth: fix `make run-vmauth` command	2020-05-22 16:45:19 +03:00
Aliaksandr Valialkin	0794cb35f2	docs/Single-server-VictoriaMetrics.md: mention about `vmauth` in `Security` section	2020-05-21 23:48:32 +03:00
Aliaksandr Valialkin	c0933ce926	docs/Cluster-VictoriaMetrics.md: mention about `vmauth` service in `Multitenancy` chapter	2020-05-21 22:53:45 +03:00
Aliaksandr Valialkin	3a3ff50548	docs/Single-server-VictoriaMetrics.md: sync with single-node README.md	2020-05-21 20:45:57 +03:00
Aliaksandr Valialkin	dcbdc009f5	app/vmagent: check for error returned from flag.Set	2020-05-21 16:30:48 +03:00
Aliaksandr Valialkin	b59e089ac7	app/vmagent: add `-dryRun` option for checking all the configs mentioned in command-line flags without running `vmagent` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:18 +03:00
Aliaksandr Valialkin	482bae8466	lib/promscrape: add `-promscrape.config.dryRun` flag for checking `-promscrape.config` for errors or unsupported options Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 14:54:32 +03:00
Aliaksandr Valialkin	901093279e	app/vmstorage/transport: update stale comment - vmstorage now sends small `ack` packets to `vminsert`	2020-05-21 14:04:52 +03:00
Aliaksandr Valialkin	a5e57a76eb	docs/vmagent.md: sync with app/vmagent/README.md	2020-05-21 12:11:22 +03:00
kreedom	2752d6cb26	vmalert add quotes escape function (#510 ) * vmalert add quotes escape function Co-authored-by: kreedom	2020-05-21 12:10:35 +03:00
Aaron France	b26245c48b	Update README.md	2020-05-21 12:10:33 +03:00
Aliaksandr Valialkin	d83c68ca03	app/vmselect/promql: add `ascent_over_time(m[d])` and `descent_over_time(m[d])` functions These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin	8ff28f5b91	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin	071d58864b	vendor: update github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 02:55:17 +03:00
faceair	504557785e	keep debug symbols (#438 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-05-20 01:23:14 +03:00
Aliaksandr Valialkin	ec6fb5a323	docs/MetricsQL.md: add a link to https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085	2020-05-20 00:51:27 +03:00
Roman Khavronenko	00a7eab43d	dashboards: update troubleshooting row (#506 ) * Slow metrics load panel was removed since it is hard to interpret without additional metrics and stats; * Slow inserts panel was updated to display percentage of slow inserts comparing to total number of inserts to show the real impact.	2020-05-20 00:51:12 +03:00
Aliaksandr Valialkin	ddc9e69bd6	docs/vmagent.md: mention an alternative to `refresh_interval` option in scrape configs	2020-05-19 23:10:16 +03:00
Aliaksandr Valialkin	73ec5cf460	lib/promscrape: add `-promscrape.discovery.concurrency` and `-promscrape.discovery.concurrentWaitTime` flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes	2020-05-19 17:35:59 +03:00
Aliaksandr Valialkin	7d46dd452a	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin	37068064dd	app/vmselect/promql: fix `outilersk` calculations	2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin	5e4d08ac22	docs/Quick-Start.md: mention that `vmagent` can be used instead of Prometheus in most cases	2020-05-19 14:09:16 +03:00
Aliaksandr Valialkin	fc81ea38d4	app/vmselect/promql: add `outliersk(N, m)` aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin	9ca781b8f0	app/vmalert/notifier: go fmt	2020-05-19 13:00:18 +03:00
Roman Khavronenko	8e29b4a716	dashboards: updates and fixes for cluster version (#500 ) * The new update introduces new row "Troubleshooting" that contains panels for churn rate and slow-queries/inserts/loads metrics. This row supposed to be reveal the cause of low performance or other issues; * CPU panel got `short` units instead of `seconds`; * Overview row was updated with panel showing bytes-per-datapoint stat; * Overview row was updated with panel showing free disk space.	2020-05-19 11:57:20 +03:00
kreedom	27911ae179	vmalert - add expr to variables, add escape functions (#495 ) * vmalert - add expr to variables, add escape functions Co-authored-by: kreedom	2020-05-19 11:55:03 +03:00
Roman Khavronenko	c7f3e58032	vmalert: avoid sending resolves for pending alerts (#498 ) Before the change we were sending notifications to notifier if following conditions are met: * alert is in Fire state * alert is in Inactive state We were sending Inactive notifications to resolve alert ASAP. Unfortunately, we were sending resolves for Pending alerts that become Inactive, which is wrong. In this change we delete alert from the active list if it was Pending and become Inactive. In this way we now have Inactive alerts only if they were in state Fire before. See test change for example.	2020-05-19 11:55:00 +03:00
Roman Khavronenko	e5f5342e18	vmalert: fix potential race during configuration reloads (#497 ) Configuration reload and rules evaluation can't be executed in same time now. This may make reload time longer but prevents from potential races.	2020-05-19 11:54:55 +03:00
Aliaksandr Valialkin	c0c6581601	docs/Articles.md: add https://www.robustperception.io/evaluating-performance-and-correctness to third-party posts	2020-05-17 00:35:30 +03:00
Aliaksandr Valialkin	32a1fa9fd3	deployment/docker: update Go builder from v1.14.2 to v1.14.3 This should fix the following issues found in Go v1.14.2. See https://github.com/golang/go/issues?q=milestone%3AGo1.14.3+label%3ACherryPickApproved for details.	2020-05-16 22:53:09 +03:00
Aliaksandr Valialkin	b99d03a956	app/vmalert: run `make quicktemplate-gen` from the root dir of the repository	2020-05-16 22:45:45 +03:00
Aliaksandr Valialkin	5fbab64b0f	docs/Single-server-VictoriaMetrics.md: put `contact us` email to the top of the page	2020-05-16 22:37:41 +03:00
Aliaksandr Valialkin	0528d3fed9	docs/Single-server-VictoriaMetrics.md: add `Replication` and `Backups` sections	2020-05-16 22:28:16 +03:00
Aliaksandr Valialkin	f3dbcb73ce	docs/Cluster-VictoriaMetrics.md: add missing endpoints to the list: `api/v1/import/csv` and `api/v1/status/tsdb`	2020-05-16 22:12:58 +03:00
Aliaksandr Valialkin	2784015a4d	all: print `--help` output to stdout instead of stderr This is easier to grep and pipe	2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin	a5a21739ac	docs/Quick-Start.md: update old link to Docker hub to new link	2020-05-16 10:23:40 +03:00
Aliaksandr Valialkin	2a8f1e6931	lib/storage: do not increment `vm_slow_metric_name_loads_total` counter for metric_ids which shouldnt be prefetched, since this may mislead users	2020-05-16 10:23:39 +03:00
Aliaksandr Valialkin	dc16cdd1ca	lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484	2020-05-16 09:32:30 +03:00
肖贝贝	c154a92d29	fix: fix vmagent multi queue may become one because sync bug (#484 ) Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-16 09:32:29 +03:00
Aliaksandr Valialkin	dbf8048134	app/vmrestore: document better that `vmrestore` works like `rsync --delete`, i.e. it deletes files in `-storageDataPath`, which are missing in the backup	2020-05-16 09:02:46 +03:00
Aliaksandr Valialkin	e544155a82	app/vmagent/Makefile: fix `make run-vmagent` rule	2020-05-15 19:35:16 +03:00
Aliaksandr Valialkin	6c43ba1cb1	app/vmagent/remotewrite: remove unused import after the commit `93267f143f`	2020-05-15 17:42:31 +03:00
Aliaksandr Valialkin	1d71253653	app/vmagent/remotewrite: allow ingesting time series with multiple samples at once Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481	2020-05-15 17:37:27 +03:00
Aliaksandr Valialkin	0f3d46810b	lib/backup: remove misleading `-dst` mention in error message Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482	2020-05-15 17:13:27 +03:00
Aliaksandr Valialkin	e72518e8c6	lib/backup: donload only the remaining parts for partially downloaded files after `vmrestore` restart Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487	2020-05-15 17:03:25 +03:00
Aliaksandr Valialkin	a853869e75	app/vmstorage/transport: prevent from uncontrolled memory usage growth when `vminsert` sends big packets with too long labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/490	2020-05-15 15:42:54 +03:00
Aliaksandr Valialkin	d1c8b0d6e9	.github/workflows: an attempt to fix loading of golangci-lint	2020-05-15 15:06:42 +03:00
Aliaksandr Valialkin	bdbb5f6cfe	vendor: `make vendor-update`	2020-05-15 15:03:11 +03:00
Aliaksandr Valialkin	1e5c1d7eaa	app/vmstorage: add `vm_slow_metric_name_loads_total` metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:12:24 +03:00
Aliaksandr Valialkin	d6b9a49481	app/vmstorage: add `vm_slow_row_inserts_total` and `vm_slow_per_day_index_inserts_total` metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:46:57 +03:00
Aliaksandr Valialkin	2c4d05db10	docs/vmalert.md: sync with app/vmalert/README.md	2020-05-15 13:26:57 +03:00
Roman Khavronenko	e850bf0eff	vmalert: fix the access to rules slice element by wrong index (#486 ) During group's update rules deletion was causing slice mutations while slice index was assumed to be unchanged. This caused "slice bounds out of range" errors when multiple rules were deleted sequentially.	2020-05-15 13:26:06 +03:00
hagen1778	d369450f27	vmalert: update README	2020-05-15 13:26:04 +03:00
Aliaksandr Valialkin	a72f18e821	lib/{storage,mergeset}: further tuning of compression levels depending on block size This should improve performance for querying newly added data, since it can be unpacked faster.	2020-05-15 13:12:28 +03:00
Aliaksandr Valialkin	2cf2e9955b	lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish	2020-05-15 12:12:20 +03:00
Aliaksandr Valialkin	67e331ac62	lib/storage: optimize ingestion pefrormance for new time series	2020-05-15 12:12:19 +03:00
Aliaksandr Valialkin	6838fa876c	lib/mergeset: tune compression levels in order to improve ingestion performance a bit	2020-05-15 12:12:15 +03:00
Aliaksandr Valialkin	1b5d272e07	lib/storage: reduce indentation in Storage.add	2020-05-14 23:23:56 +03:00
Aliaksandr Valialkin	71d29a8fa1	lib/storage: return the first error instead of the last error, since the first error usually points to the root cause	2020-05-14 23:18:59 +03:00
Aliaksandr Valialkin	3845420a8f	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin	7e831741f9	lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused	2020-05-14 15:32:23 +03:00
Aliaksandr Valialkin	2f42b85e0e	lib/storage: document that getnerateUniqueMetricID should return dense ids	2020-05-14 14:08:59 +03:00
Aliaksandr Valialkin	f442d81648	lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse	2020-05-14 14:03:15 +03:00
Aliaksandr Valialkin	4bc3d284fa	docs/vmalert.md: sync with app/vmalert/README.md	2020-05-13 22:57:29 +03:00
Roman Khavronenko	e208e76222	vmalert: check if remoteRead object was initied before calling Restore (#473 ) The check for non-nil remoteRead was mistakenly dropped during refactoring which caused panics when `vmalert` wasn't configured with `remoteRead` flag.	2020-05-13 22:57:26 +03:00
Roman Khavronenko	1523890742	vmalert: fix flag names and description in README (#475 ) Change also adds the recommendation for `remotewrite` queue error.	2020-05-13 22:57:20 +03:00
肖贝贝	8c3e9adf7f	Feat/vmalert add max queue size (#472 ) * feat: add remoteWrite.maxQueueSize to reduce queue full * rename remote(write\|read) flags to remote(Write\|Read) for the sake of consistency Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-13 22:57:16 +03:00
Aliaksandr Valialkin	bac9a684e8	docs/vmbackup.md: add a link to vmbackuper tool	2020-05-13 22:57:11 +03:00
Aliaksandr Valialkin	f3d9a5b0ec	app/vmselect/promql: suppress "SA4006: this value of `dstValues` is never used" error in golangci-lint	2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin	8bb44a5d09	lib/storage: optimize label matching for regexp ending with literal suffix For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`, so it should work faster now.	2020-05-13 11:39:05 +03:00
Aliaksandr Valialkin	3b0f66a227	app/vmagent: fix a bug with improper relabeling when multiple `-remoteWrite.urlRelableConfig` args are set This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:03:45 +03:00
Aliaksandr Valialkin	18a0caee43	app/vmselect/promql: fix `any(..)` calculations - return all the data points instead of the first one	2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin	3d3f41b961	app/vmstorage/transport: fix panic during server stop on 32-bit arches See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2020-05-12 20:21:40 +03:00
Aliaksandr Valialkin	c9ab6dc532	lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM	2020-05-12 20:21:39 +03:00
Aliaksandr Valialkin	81b8811cf4	app/vmselect/promql: remove `-search.maxPointsPerTimeseries` command-line flag Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin	408ade27a9	app/vmselect/promql: add `any(x) by (y)` aggregate function, which returns any time series from `q` for each group `y`	2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin	21c2982ac8	app/vmselect/promql: support for `sum(x) by (y) limit N` syntax in order to limit the number of output time series after aggregation	2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin	f341c6fcc4	Revert "app/vmselect: add `-search.estimatedSeriesCountAfterAggregation` command-line flag for tuning the probability of OOMs or false-positive `not enough memory` errors" This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c. Reason for revert: this commit has been removed from single-node version	2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin	d54a93fc81	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:43 +03:00
Aliaksandr Valialkin	405cf44aed	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:53 +03:00
Aliaksandr Valialkin	da6a84e147	app/vmagent/remotewrite: properly dial TCP6 addresses set via `-remoteWrite.url` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469	2020-05-12 15:24:50 +03:00
Aliaksandr Valialkin	bd5f4e0344	lib/storage: properly initialize part struct before trying to close it on error This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .	2020-05-12 14:54:16 +03:00
Aliaksandr Valialkin	cc825c483b	vendor: `make vendor-update`	2020-05-12 14:26:29 +03:00
Aliaksandr Valialkin	ddd8c9d099	deployment/docker: omit http2 support in `*-prod` binaries VictoriaMetrics doesn't use http/2.0, so disable it completely. Use `nethttpomithttp2` tag defined in Go1.14 for this. See `2566e21f24` for details.	2020-05-12 14:19:33 +03:00
Aliaksandr Valialkin	4e237b4670	app/vminsert/influx: support passing AccountID and ProjectID via plain TCP and UDP Now `vminsert` accepts AccountID and ProjectID via `VictoriaMetrics_AccountID` and `VictoriaMetrics_ProjectID` tags when reading Influx line protocol data via plain TCP or UDP (i.e. when `-influxListenAddr` is set).	2020-05-12 13:13:04 +03:00
Aliaksandr Valialkin	f7753b1469	lib/storage: gradually pre-populate per-day inverted index for the next day This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:32 +03:00
Aliaksandr Valialkin	8c77cb436a	lib/storage: typo fixes in error messages: `or -> of`	2020-05-12 12:12:33 +03:00
Aliaksandr Valialkin	bbf06a4248	lib/storage: speed up matching for common regexps in label filters The following regexps have been optimized: * 'foo.+bar' * 'foo.+bar.+baz' This should improve performance for matching Graphite-like metrics.	2020-05-11 22:49:01 +03:00
Aliaksandr Valialkin	37254a139a	lib/storage: add a benchmark for Graphite-like regexps for metric names	2020-05-11 22:49:00 +03:00
Roman Khavronenko	0157566fdb	vmalert: cleanup and restructure of code to improve maintainability (#471 ) The change introduces new entity `manager` which replaces `watchdog`, decouples requestHandler and groups. Manager supposed to control life cycle of groups, rules and config reloads. Groups export an ID method which returns a hash from filename and group name. ID supposed to be unique identifier across all loaded groups. Some tests were added to improve coverage. Bug with wrong annotation value if $value is used in templates after metrics being restored fixed. Notifier interface was extended to accept context. New set of metrics was introduced for config reload.	2020-05-11 14:35:55 +03:00
Nikolay Khramchikhin	0e8c345ffb	vmalert config reload added config hot reload for vmalert with sighup and api call	2020-05-11 14:35:50 +03:00
Aliaksandr Valialkin	6ce9f81d16	docs/CaseStudies.md: add CERN case study	2020-05-11 14:35:43 +03:00
Aliaksandr Valialkin	6c88e3523b	docs/Single-server-VictoriaMetrics.md: small updates for `Monitoring` and `How to start VictoriaMetrics` sections	2020-05-08 20:35:31 +03:00
Aliaksandr Valialkin	6646b380ef	docs/vmauth.md: fix a link to docker images	2020-05-08 14:11:10 +03:00
Aliaksandr Valialkin	0362bd220e	docs/Articles.md: add a link to CERN article at https://indico.cern.ch/event/877333/contributions/3696707/attachments/1972189/3281133/CMS_mon_RD_for_opInt.pdf	2020-05-08 01:25:17 +03:00
Aliaksandr Valialkin	657c3e3fc5	Makefile: suppress false positives for golangci-lint on nil pointer dereference	2020-05-07 19:41:11 +03:00
Aliaksandr Valialkin	28ad350a31	app/vmagent: return 200 from `/-/reload` endpoint as Prometheus does	2020-05-07 19:29:48 +03:00
Aliaksandr Valialkin	2f28e945b8	lib/httpserver: add `-http.shutdownDelay` flag for a grace period before http server shutdown The http server returns 503 non-OK error at `/health` page during grace period, so load balancers in front of the http server could re-route incoming requests to other servers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/463	2020-05-07 15:25:51 +03:00
Aliaksandr Valialkin	3052b479b7	lib/httpserver: reduce typical duration for http server graceful shutdown Previously the duration for graceful shutdown for http server could take more than a minute because of imporperly set timeouts in setNetworkTimeout. Now typical duration for graceful shutdown should be reduced to less than 5 seconds.	2020-05-07 14:16:38 +03:00
Aliaksandr Valialkin	dc04040781	docs/{vmagent,vmauth}: small clarifications in the docs	2020-05-07 12:55:06 +03:00
Aliaksandr Valialkin	2b403d3f42	app/vmauth: prevent from attacks with `..` in path for accessing resources outside the configured `url_prefix`	2020-05-07 12:55:04 +03:00
Aliaksandr Valialkin	c43a265716	lib/flagutil: make errcheck happy by explicitly ignoring Array.Set result in tests	2020-05-06 22:37:28 +03:00
Aliaksandr Valialkin	15e3682b40	lib/flagutil: properly parse quoted flag values for flagutil.Array	2020-05-06 22:28:15 +03:00
Aliaksandr Valialkin	20538a2a5d	app/vmagent: allow setting independent auth configs per each configured `-remoteWrite.url`	2020-05-06 16:52:32 +03:00
Aliaksandr Valialkin	12dbb9e22c	app/vmagent: properly set client-side TLS certificates for `-remoteWrite.url`. Previously they were mistakenly set as server-side	2020-05-06 16:50:37 +03:00
Aliaksandr Valialkin	9f39e618ed	lib/promscrape/discovery/gce: discover per-zone instances for `gce_sd_config` in parallel. This should reduce discovery latency	2020-05-06 15:00:23 +03:00
Aliaksandr Valialkin	8665c2edb1	docs/vmagent.md: small fixes	2020-05-06 14:49:25 +03:00
Aliaksandr Valialkin	8ab5e47b5c	lib/promscrape: add Prometheus-compatible DNS-based service discovery aka `dns_sd_configs`	2020-05-06 00:02:41 +03:00
Aliaksandr Valialkin	42d563934b	lib/promscrape: properly connect to TCP6 addresses if `-enableTCP6` is set	2020-05-06 00:02:40 +03:00
Aliaksandr Valialkin	21b91599c2	docs/{vmauth,vmagent}: fix ports for profiling	2020-05-05 20:16:09 +03:00
Aliaksandr Valialkin	309700ab8c	docs/vmauth.md: mention that we can help creating customized proxy	2020-05-05 12:34:08 +03:00
Aliaksandr Valialkin	20e958789a	docs/{vmagent,vmauth}: add `Profiling` section	2020-05-05 11:45:29 +03:00
Aliaksandr Valialkin	1153f30fee	docs: add vmauth.md	2020-05-05 11:17:45 +03:00
Aliaksandr Valialkin	782fb30cd0	app/vmauth: build fixes	2020-05-05 11:03:25 +03:00
Aliaksandr Valialkin	de31d16154	app/vmauth: add initial version of vmauth. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details	2020-05-05 10:56:20 +03:00
Aliaksandr Valialkin	61df59b9ea	docs/vmagent.md: `/targets` page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead	2020-05-05 10:56:18 +03:00
Aliaksandr Valialkin	1c8e97c8a0	lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP	2020-05-05 10:56:15 +03:00
Aliaksandr Valialkin	dde92fccc5	docs/vmalert.md: sync with app/vmalert/README.md	2020-05-05 07:51:32 +03:00
Aliaksandr Valialkin	054457d1f4	lib/promscrape: allow explicitly setting empty token via `token: ""` in `consul_sd_config`	2020-05-05 07:49:54 +03:00
Aliaksandr Valialkin	fd739808f3	`make vendor-update`	2020-05-05 00:53:41 +03:00
Roman Khavronenko	abce2b092f	app/vmalert: restore alerts state from datasource metrics (#461 ) * app/vmalert: restore alerts state from datasource metrics Vmalert will restore alerts state for rules that have `rule.For` > 0 from previously written timeseries via `remotewrite.url` flag. * app/vmalert: mention remotewerite and remoteread configuration in README	2020-05-05 00:52:19 +03:00
Aliaksandr Valialkin	89aa6dbf56	lib/promscrape: add Prometheus-compatible service discovery for Consul aka `consul_sd_configs` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-05-04 20:53:06 +03:00
Aliaksandr Valialkin	28e0e8fd88	lib/promauth: properly set up client certificate in tls.Config Previously the client certificate has been mistakenly set up as a server certificate	2020-05-04 20:53:04 +03:00
Aliaksandr Valialkin	ed91fe1d9b	lib/promscrape: move common code for discovery api config map handling into discoveryutils	2020-05-04 20:52:58 +03:00
Aliaksandr Valialkin	c50fd219dc	lib/promscrape/discovery/kubernetes/: unify apiConfig creation	2020-05-04 20:52:53 +03:00
Aliaksandr Valialkin	54414fefef	vendor: update github.com/valyala/quicktemplate from v1.4.1 to v1.5.0	2020-05-04 01:37:34 +03:00
Aliaksandr Valialkin	6606dff58d	docs/Single-server-VictoriaMetrics.md: mention that it is recommended upgrading to the latest release before reporting issues	2020-05-04 00:42:33 +03:00
Aliaksandr Valialkin	e3a4b75e59	docs/Cluster-VictoriaMetrics.md: add `Multitenancy` chapter	2020-05-03 18:01:15 +03:00
Aliaksandr Valialkin	a5880f17af	lib/promscrape: remove debug line left after the commit `e4aac6ea40`	2020-05-03 17:16:19 +03:00
Aliaksandr Valialkin	1f0e8fdc0d	lib/promscrape: fix tests after the commit `658a8742ac` The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does. See https://www.robustperception.io/life-of-a-label for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453	2020-05-03 16:59:29 +03:00
DexterZhang	317688f144	fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] (#454 )	2020-05-03 16:59:28 +03:00
Aliaksandr Valialkin	ab1e6a76bb	lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels This should make consistent intervals between data points for scrape targets across reloads. Previously these intervals were random.	2020-05-03 14:31:22 +03:00
Aliaksandr Valialkin	f25416984b	lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after `3b234d82e5`	2020-05-03 14:31:20 +03:00
Aliaksandr Valialkin	f422203e10	lib/promscrape: reload only modified scrapers on config changes This should improve scrape stability when big number of targets are scraped and these targets are frequently changed. Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests: - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460	2020-05-03 12:47:16 +03:00
Aliaksandr Valialkin	8f591b848a	docs/MetricsQL.md: document `first_over_time` and `last_over_time` functions	2020-05-03 12:47:16 +03:00
Aleksey Shirokih	137e371219	Avoid ugly y-label for rows inserted (#457 )	2020-05-02 19:06:37 +01:00
Aliaksandr Valialkin	bbaca16ce8	lib/httpserver: rename `http.externalURL` to `http.pathPrefix` and improve help message for this flag The `http.externalURL` flag name was slightly misleading, so it has been renamed to `http.pathPrefix`.	2020-05-02 13:12:24 +03:00
DexterZhang	a0589f2ca5	feat(httpserver): add `http.externalUrl` config to http server, it adds prefix to http path automatically (#452 )	2020-05-02 13:12:23 +03:00
Aliaksandr Valialkin	8e041f1911	docs/Single-server-VictoriaMetrics.md: hint that `\n` is a single newline char	2020-05-01 13:42:50 +03:00
Aliaksandr Valialkin	b21b73115a	app/vminsert: add `/-/reload` handler in the same way as for `vmagent`	2020-04-30 02:18:08 +03:00
Aliaksandr Valialkin	a970705d8e	lib/procutil: prevent from app termination on SIGHUP signal, since this signal is frequently used for config reload	2020-04-30 02:18:06 +03:00
DexterZhang	ae215e5538	feat(vmagent): add promscrap config reload suppport via http (#450 ) * feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload` * fix: typo fix	2020-04-30 02:18:01 +03:00
Aliaksandr Valialkin	d99f48aa48	lib/httpserver: mention that `-http.maxGracefulShutdownDuration` command-line flag value can be increased on shutdown timeout	2020-04-30 01:37:02 +03:00
Aliaksandr Valialkin	fbfa6aa9f0	docs/Single-server-VictoriaMetrics.md: mention that it is better to increase CPU and RAM per `vmselect` node in order to achieve higher query performance	2020-04-30 00:53:14 +03:00
Aliaksandr Valialkin	c19f67a248	docs: add vmalert.md	2020-04-29 17:42:16 +03:00
Artem Navoiev	121f7e1d56	Update README.md	2020-04-29 17:41:04 +03:00
Aliaksandr Valialkin	15876c6425	docs/Single-server-VictoriaMetrics.md: update `Alerting` section	2020-04-29 17:39:56 +03:00
Aliaksandr Valialkin	de5f923476	lib/promscrape: set 30 seconds timeout for discovery api requests Previously such requests could hang for long time. This could make debugging harder.	2020-04-29 17:29:03 +03:00
Aliaksandr Valialkin	b6d88bac04	vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp The upstream fasthttp may contain issues like `996610f021` , plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.	2020-04-29 16:43:09 +03:00
Aliaksandr Valialkin	473188f4fd	docs/Single-server-VictoriaMetrics.md: mention that basic downsampling could be made with the help of de-duplication	2020-04-28 16:39:06 +03:00
Aliaksandr Valialkin	9ed4951ec8	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin	cd1145e5f4	app/vmselect: add `-search.estimatedSeriesCountAfterAggregation` command-line flag for tuning the probability of OOMs or false-positive `not enough memory` errors	2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin	d78ed50edd	lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet. Log about this and delete the corresponding metricID, so it could be re-created next time.	2020-04-28 12:01:32 +03:00
Aliaksandr Valialkin	a858b7e393	app/vmalert: added missing comments for public entities	2020-04-28 11:19:48 +03:00
Aliaksandr Valialkin	716bbe79d4	app/vminsert/netstorage: increase timeout for waiting for `ack` message after sending big data block to vmstorage	2020-04-28 11:19:46 +03:00
Aliaksandr Valialkin	d435029d10	docs/Articles.md: add https://zerodha.tech/blog/infra-monitoring-at-zerodha/	2020-04-28 02:24:36 +03:00
Aliaksandr Valialkin	53740d0026	lib/promscrape: handle connection reset when targets responds with http redirect	2020-04-28 02:14:32 +03:00
肖贝贝	3e6f29f462	fix: vmagent not follow 301/302 redirect bug (#445 ) Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-28 02:14:31 +03:00
Aliaksandr Valialkin	424068f804	lib/promscrape: handle connection reset when targets responds with http redirect	2020-04-28 02:14:26 +03:00
肖贝贝	7d045bf2ca	fix: vmagent not follow 301/302 redirect bug (#445 ) Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-28 02:14:25 +03:00
Aliaksandr Valialkin	50af16baf2	app/vmalert: fix build	2020-04-28 00:34:01 +03:00
Aliaksandr Valialkin	e3db2c73a6	app/vmalert: sync with master branch	2020-04-28 00:19:42 +03:00
Aliaksandr Valialkin	7644f40763	app/vmalert: include it into the next release	2020-04-28 00:11:41 +03:00
Aliaksandr Valialkin	2aecf7c37c	lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting	2020-04-28 00:02:19 +03:00
Aliaksandr Valialkin	806dc73d8a	lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize	2020-04-28 00:02:18 +03:00
Aliaksandr Valialkin	a603a15757	lib/promscrape/discovery/gce: make golangci-lint happy	2020-04-27 19:29:42 +03:00
Aliaksandr Valialkin	86a1d9cb0c	lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka `ec2_sd_configs`	2020-04-27 19:29:22 +03:00
Aliaksandr Valialkin	1acb6eb25a	lib/promscrape/discovery/gce: properly set `filter` query arg in api url	2020-04-27 16:01:53 +03:00
Aliaksandr Valialkin	0daa37fa02	lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config	2020-04-27 11:45:45 +03:00
Aliaksandr Valialkin	989d84cf3f	app/{vminsert,vmstorage}: wait for `ack` from `vmstorage` after each packet sent to it from `vminsert` This should protect from possible data loss when `vmstorage` is stopped while the packet is sent from `vminsert`. This commit switches to new protocol between vminsert and vmstorage, which is incompatible with the previous protocol. So it is required that both vminsert and vmstorage nodes are updated.	2020-04-27 09:53:26 +03:00
Aliaksandr Valialkin	e933cbac16	lib/storage: postpone reading data from blocks during search This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin	23a310cc68	app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks This should minimize the number of disk seeks when reading data from temporary file.	2020-04-26 16:46:17 +03:00
Aliaksandr Valialkin	31861c5b8e	lib/promscrape/discovery/gce: allow empty `zone` arg in `gce_sd_config` - in this case zones for the given project are automatically discovered	2020-04-26 14:37:38 +03:00
Aliaksandr Valialkin	b16e19c053	lib/storage/dedup.go: go fmt	2020-04-26 14:37:36 +03:00
Aliaksandr Valialkin	a0000c3a6e	lib/storage: improve deduplication algorithm Now it leaves only the first data point on each `-dedup.minScrapeInterval` interval. Previously it may leave two data points on the interval. This could lead to unexpected results for `histogram_quantile(phi, sum(rate(buckets)) by (le))` query.	2020-04-26 13:10:18 +03:00
Aliaksandr Valialkin	d9bdda408c	docs/{vmbackup,vmrestore}.md: update `-help` output	2020-04-24 22:44:45 +03:00
Jason Gardner	7a6b2839b4	app/vmbackup: added ability to create and delete snapshots during backup (#428 ) * app/vmbackup: added ability to create and delete snapshots during backup Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422 * Add snapshot create and delete url flags * Fixed errcheck warnings in build	2020-04-24 22:35:50 +03:00
Aliaksandr Valialkin	13b4069c59	lib/storage: postpone label filters matching too many time series instead of giving up with error This should reduce the frequency of the following errors: cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range	2020-04-24 21:18:52 +03:00
Aliaksandr Valialkin	9b386e594f	docs/Single-server-VictoriaMetrics.md: document `-search.resetCacheAuthKey`	2020-04-24 19:48:13 +03:00
Aliaksandr Valialkin	32b3f959fc	app/vmselect: fix description for `-search.resetCacheAuthKey`	2020-04-24 19:44:35 +03:00
Aliaksandr Valialkin	7c74efd640	lib/promscrape/discovery/gce: make golint happy by ignoring resp.Body.Close() result	2020-04-24 18:13:26 +03:00
Aliaksandr Valialkin	987fcce93d	.github/workflows: install dependencies before code checkout Othwerise dependencies' install mangles go.mod	2020-04-24 17:55:53 +03:00
Aliaksandr Valialkin	069690e3bd	lib/promscrape: initial implementation for `gce_sd_configs` aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:53:43 +03:00
Aliaksandr Valialkin	cf68c5f66a	.github/workflows: enable Go modules when installing dependencies Disabled Go modules broke golangci-lint build	2020-04-24 17:40:43 +03:00
Aliaksandr Valialkin	c53fd515fe	docs/Single-server-VictoriaMetrics.md: mention that `-search.maxStalenessInterval` can be useful for InfluxDB and TimescaleDB users	2020-04-24 16:23:33 +03:00
Aliaksandr Valialkin	48320cffe0	.github/workflows: install golangci-lint at `Dependencies` step	2020-04-24 15:37:55 +03:00
Aliaksandr Valialkin	de7887fbf4	.github/workflows: update Go version in actions/setup-go from v1.13 to v1.14	2020-04-24 15:31:12 +03:00
Aliaksandr Valialkin	c66daf1f0a	vendor: `make vendor-update`	2020-04-24 15:28:37 +03:00
Aliaksandr Valialkin	8d76795be5	.github/workflows: use master branch for 'actions/setup-go' and 'actions/checkout'	2020-04-24 14:42:06 +03:00
Aliaksandr Valialkin	de991551f5	lib/promscrape: query `/api/v1/namespaces/*` for the configured namespaces in `kubernetes_sd_config` This should fix authroization issues described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432	2020-04-24 14:42:02 +03:00
Aliaksandr Valialkin	387a21c96d	lib/promscrape: add `-promscrape.configCheckInterval` command-line flag for automating config checking Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/431	2020-04-23 23:41:26 +03:00
Aliaksandr Valialkin	83e4c8427e	lib/promscrape: access Config entries by reference, so they can be compared by addresses	2020-04-23 14:38:29 +03:00
Aliaksandr Valialkin	a5ad19e836	vendor: update google.golang.org/api from v0.21.0 to v0.22.0	2020-04-23 14:30:58 +03:00
Aliaksandr Valialkin	b0f6d3244c	vendor: update github.com/aws/aws-sdk-go from v1.30.8 to v1.30.12	2020-04-23 12:36:14 +03:00
Aliaksandr Valialkin	e220f3eeb6	lib/promscrape: move KubernetesSDConfig to lib/promscrape/discovery/kubernetes	2020-04-23 11:34:30 +03:00
Aliaksandr Valialkin	1187494c8f	lib/promscrape/discovery/kubernetes: hide role switch logic behind GetLabels function	2020-04-22 22:16:18 +03:00
Aliaksandr Valialkin	f9526809e5	app/vmselect: add `/api/v1/status/tsdb` page with useful stats for locating root cause for high cardinality issues See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin	36f6935ddd	vendor: update github.com/valyala/fastjson from v1.5.0 to v1.5.1	2020-04-21 00:04:14 +03:00
Aliaksandr Valialkin	76c4140da7	vendor: update github.com/valyala/gozstd from v1.6.4 to v1.7.0	2020-04-20 23:04:20 +03:00
Aliaksandr Valialkin	f3e5722257	lib/writeconcurrencylimiter: improve docs for -maxConcurrentInserts command-line flag	2020-04-20 21:03:09 +03:00
Aliaksandr Valialkin	b59f1f1504	app/vmselect: add `-search.minStalenessInterval` command-line flag for removing gaps on graphs built from time series with irregular duration between samples Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin	603d4c9217	app/vmselect: merge `-search.maxLookback` and `-search.maxStalenessInterval` flags, since it has been appeared they have identical purpose :( Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin	82b2524f28	deployment/docker/docker-compose.yml: bump Prometheus from v2.17.1 to v2.17.2 and Grafana from v6.7.1 to v6.7.2	2020-04-20 17:30:22 +03:00
Aliaksandr Valialkin	81481abaa9	lib/promscrape/discovery/kubernetes: reuse a client for empty `api_server` inside different jobs	2020-04-20 17:07:37 +03:00
Aliaksandr Valialkin	d5b38eeac4	docs/Single-server-VictoriaMetrics.md: mention about vmagent in the end of `Prometheus setup` section	2020-04-20 16:42:50 +03:00
Aliaksandr Valialkin	db5fe03170	deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var For example, the following command will build VictoriaMetrics docker image on top of alpine image: ROOT_IMAGE=alpine make package-victoria-metrics	2020-04-20 01:16:21 +03:00
Aliaksandr Valialkin	e6277165af	deployment/docker/base: remove unused group and passwd files	2020-04-19 23:31:44 +03:00
Aliaksandr Valialkin	57311d748d	Makefile: increase the timeout for `make golangci-lint` from 1 minute to 2 minutes This should fix timeout errors on GitHub actions	2020-04-17 19:13:57 +03:00
Aliaksandr Valialkin	1b911f6965	app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection This should fix the following error when sending data to remote storage: couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-17 15:53:17 +03:00
Aliaksandr Valialkin	6764efde39	lib/promscrape/discovery/kubernetes: update stale comments	2020-04-17 14:06:26 +03:00
Aliaksandr Valialkin	da05904638	vendor: `make vendor-update`	2020-04-17 13:25:18 +03:00
Aliaksandr Valialkin	9105f72f17	docs/vmagent.md: typo fix: unvailable -> unavailable	2020-04-17 13:12:13 +03:00
Aliaksandr Valialkin	d46311fd93	app/vmagent/README.md: mention about `prodmscrape.suppressScrapeErrors`	2020-04-17 13:09:08 +03:00
Aliaksandr Valialkin	b9b5641c2f	app/vmselect: properly apply `-search.maxLookback` to queries sent to `/api/v1/query`	2020-04-17 12:31:18 +03:00
Dmitry Shihovtsev	41bb31ecf6	Fix misspelled Cortex name in the FAQ (#421 )	2020-04-17 12:31:15 +03:00
Aliaksandr Valialkin	d86640d609	lib/promscrape: suppress scrape errors if `-promscrape.suppressScrapeErrors` flag is set	2020-04-16 23:41:52 +03:00
Aliaksandr Valialkin	70104f3fb1	lib/promscrape: print all the labels for the target on error message for failed scrape This should improve debuggability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/420	2020-04-16 23:35:10 +03:00
Aliaksandr Valialkin	266bbec52d	lib/promscrape: retry target scraping when the target closes previously established keep-alive connection to it This should fix the following error: the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-16 23:25:34 +03:00
Aliaksandr Valialkin	e2c3e1d2e5	docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics supports Kubernetes service discovery	2020-04-16 18:40:51 +03:00
Aliaksandr Valialkin	a22a2e9bf4	docs/Single-server-VictoriaMetrics.md: typo fix: unneded -> unneeded	2020-04-16 17:35:47 +03:00
Aliaksandr Valialkin	71c122a814	docs/Single-server-VictoriaMetrics.md: imrpove docs about metrics deletion	2020-04-16 17:32:44 +03:00
Aliaksandr Valialkin	30baf65aa7	docs/Single-server-VictoriaMetrics.md: mention that the `delete API` can be protected by `authKey`	2020-04-16 17:20:25 +03:00
Aliaksandr Valialkin	b2d009c8db	lib/logger: typo fix	2020-04-16 00:20:02 +03:00
Aliaksandr Valialkin	d4bc60d63c	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:45 +03:00
Aliaksandr Valialkin	d23a8b7462	docs/Single-server-VictoriaMetrics.md: typo fix	2020-04-15 15:22:36 +03:00
Aliaksandr Valialkin	9fd1827824	vendor: `make vendor-update`	2020-04-15 14:51:51 +03:00
Aliaksandr Valialkin	1d4afde6a9	docs/Single-server-VictoriaMetrics.md: clarify how to use `-influxListenAddr` command-line option	2020-04-15 12:34:32 +03:00
Aliaksandr Valialkin	a873b553cf	app/vmselect: handle `timestamp(metric offset X)` the same way as Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin	99f0cb1f5f	lib/promscrape: code cleanup in runScraper func	2020-04-15 11:36:35 +03:00
Aliaksandr Valialkin	90bd92a6f7	docs/Single-server-VictoriaMetrics.md: mention that backfilling can be done via any supported ingestion method	2020-04-15 10:57:26 +03:00
Aliaksandr Valialkin	e9d9638627	lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet. This should fix the following error: error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF	2020-04-15 00:10:11 +03:00
Aliaksandr Valialkin	2ce78c0dde	docs/Single-server-VictoriaMetrics.md: add https://github.com/Slapper/ansible-victoriametrics-cluster-role to `integrations` chapter	2020-04-14 16:28:12 +03:00
Aliaksandr Valialkin	6ec582acb9	lib/promscrape: show information on improperly configured scrape targets at the bottom of `/targets` page This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:13 +03:00
Aliaksandr Valialkin	391fb0903e	lib/promscrape/discovery/kubernetes: remove only unused client for API server during cleaning	2020-04-14 14:19:26 +03:00
Aliaksandr Valialkin	636e1578de	lib/promscrape: add promrelabel.GetLabelValueByName helper function	2020-04-14 14:12:15 +03:00
Aliaksandr Valialkin	3945bf9dec	lib/promscrape: mention job name in error messages when target cannot be scraped This should improve debuggability	2020-04-14 13:33:18 +03:00
Aliaksandr Valialkin	66da177fe9	lib/promscrape: reset ScrapeWork.ID in tests	2020-04-14 13:31:37 +03:00
Aliaksandr Valialkin	88366cad15	lib/promscrape: properly expose statuses for targets with duplicate scrape urls at `/targets` page Previously targets with duplicate scrape urls were merged into a single line on the page. Now each target with duplicate scrape url is displayed on a separate line.	2020-04-14 13:10:06 +03:00
Aliaksandr Valialkin	09f796e2ab	lib/promscrape: remove labels starting with `__meta_` after applying `relabel_configs` as Prometheus does This should reduce CPU load during scraping when target discovery generates big number of `__meta_*` labels (for instance, k8s discovery). See https://www.robustperception.io/life-of-a-label for details.	2020-04-14 12:23:30 +03:00
Aliaksandr Valialkin	f58d15f27c	lib/promscrape: rename 'scrape_config->scrape_limit' to 'scrape_config->sample_limit' `scrape_config` block from Prometheus config contains `sample_limit` field, while in `vmagent` this field was mistakenly named as `scrape_limit`.	2020-04-14 12:00:03 +03:00
Aliaksandr Valialkin	755f649c72	docs/vmagent.md: mention that vmagent supports `kubernetes_sd_configs` now	2020-04-13 21:07:00 +03:00
Aliaksandr Valialkin	7c4fb038e3	lib/promscrape: add initial support for `kubernetes_sd_config` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334	2020-04-13 21:03:53 +03:00
Aliaksandr Valialkin	4017163393	lib/promscrape: add `-promscrape.config.strictParse` flag for detecting errors in `-promscrape.config` file	2020-04-13 13:15:52 +03:00
Aliaksandr Valialkin	7fbfef2aee	lib/promscrape: extract common auth code to lib/promauth	2020-04-13 12:59:22 +03:00
Aliaksandr Valialkin	1ce6c311dd	vendor: `make vendor-update`	2020-04-10 18:42:02 +03:00
Aliaksandr Valialkin	e12c97f0b7	vendor: update github.com/klauspost/compress from v1.10.3 to v1.10.4	2020-04-10 18:38:50 +03:00
Aliaksandr Valialkin	3f417ce4d8	deployment/docker: update Go builder image from go1.14.1 to go1.14.2	2020-04-10 18:19:54 +03:00
Aliaksandr Valialkin	e0c6da8e2a	lib/storage: disable deduplication after dedup tests are complete The rest of tests expect that the de-duplication is disabled.	2020-04-10 17:33:38 +03:00
Aliaksandr Valialkin	8ed0d5471a	lib/storage: correctly handle `-dedup.minScrapeInterval` values smaller than 8ms Such small values may be used for removing samples with duplicate timestamps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.	2020-04-10 16:40:41 +03:00
Aliaksandr Valialkin	0b2f678d8e	lib/{storage,mergeset}: make sure that `requests` and `misses` cache counters never go down	2020-04-10 14:44:52 +03:00
Aliaksandr Valialkin	661cfb03e2	lib/protoparser: add `-*TrimTimstamp` command-line flags for Influx, Graphite, OpenTSDB and CSV data These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols	2020-04-10 12:44:46 +03:00
Aliaksandr Valialkin	f0b08dbd9e	lib/workingsetcache: accumulate stat counters on cache rotation This should prevent from cache stats counters going down after cache rotation, which may corrupt `cache hit ratio` graph on the official Grafan dasbhoards when using the following query: 1 - (sum(rate(vm_cache_misses_total[5m])) by (type) / sum(rate(vm_cache_requests_total[5m])) by (type))	2020-04-10 11:51:47 +03:00
Aliaksandr Valialkin	28c65b58a2	lib/memory: add more details to `-memory.allowedPercent` help message	2020-04-09 15:34:21 +03:00
Aliaksandr Valialkin	38256bd66d	docs: update minimum supported Go version from 1.12 to 1.13	2020-04-07 13:39:15 +03:00
Aliaksandr Valialkin	f5121d1e5f	docs/CaseStudies.md: updated ARNES numbers	2020-04-06 16:20:44 +03:00
Aliaksandr Valialkin	65ba430632	docs/CaseStudies.md: prettifying of the formatting	2020-04-06 15:24:57 +03:00
Aliaksandr Valialkin	d278e8e1b6	docs/CaseStudies.md: add ARNES case study	2020-04-06 15:18:31 +03:00
Aliaksandr Valialkin	5f679a0f24	docs/Single-server-VictoriaMetrics.md: cosmetic fixes in `Importing CSV data` chapter	2020-04-06 12:30:16 +03:00
Aliaksandr Valialkin	4661fa5b34	docs/FAQ.md: small fixes	2020-04-05 13:52:55 +03:00
Aliaksandr Valialkin	0452cb21ee	docs/FAQ.md: add more articles about VictoriaMetrics performance	2020-04-05 13:47:38 +03:00
Aliaksandr Valialkin	3656d0b13a	docs/Articles.md: added a link to https://www.iunera.com/kraken/fabric/time-series-database/	2020-04-04 16:40:41 +03:00
Aliaksandr Valialkin	2b4d3effad	app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request This header is required by Cortex (and, probably, other remote storage systems). See `9c1f44d090/docs/apis.md (remote-api)` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399	2020-04-04 16:24:47 +03:00
Aliaksandr Valialkin	87da127fbf	app/victoria-metrics: remove accidentally added testdata for single-node VM	2020-04-04 16:09:08 +03:00
Aliaksandr Valialkin	a012f6fe70	app/vmselect/promql: keep metric name after applying `first_over_time` and `last_over_time` functions	2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin	a53e332a93	app/vmstorage: add missing shutdown for http server on graceful shutdown This could result in the following panic during graceful shutdown when `/metrics` page is requested: http: panic serving 10.101.66.5:57366: runtime error: invalid memory address or nil pointer dereference goroutine 2050 [running]: net/http.(conn).serve.func1(0xc00ef22000) net/http/server.go:1772 +0x139 panic(0xa0fc00, 0xe91d80) runtime/panic.go:973 +0x3e3 github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache.(Cache).UpdateStats(0x0, 0xc0000516c8) github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache/cache.go:224 +0x37 github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(indexDB).UpdateMetrics(0xc00b931d00, 0xc02c41acf8) github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:258 +0x9f github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(Storage).UpdateMetrics(0xc0000bc7e0, 0xc02c41ac00) github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:413 +0x4c5 main.registerStorageMetrics.func1(0x0) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:186 +0xd9 main.registerStorageMetrics.func3(0xc00008c380) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:196 +0x26 main.registerStorageMetrics.func7(0xc) github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:211 +0x26 github.com/VictoriaMetrics/metrics.(Gauge).marshalTo(0xc000010148, 0xaa407d, 0x20, 0xb50d60, 0xc005319890) github.com/VictoriaMetrics/metrics@v1.11.2/gauge.go:38 +0x3f github.com/VictoriaMetrics/metrics.(Set).WritePrometheus(0xc000084300, 0x7fd56809c940, 0xc005319860) github.com/VictoriaMetrics/metrics@v1.11.2/set.go:51 +0x1e1 github.com/VictoriaMetrics/metrics.WritePrometheus(0x7fd56809c940, 0xc005319860, 0xa16f01) github.com/VictoriaMetrics/metrics@v1.11.2/metrics.go:42 +0x41 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.writePrometheusMetrics(0x7fd56809c940, 0xc005319860) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/metrics.go:16 +0x44 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb5a120, 0xc005319860, 0xc005018f00, 0xc00002cc90) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:154 +0x58d github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb5a120, 0xc005319860, 0xc005018f00) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:119 +0x8e net/http.HandlerFunc.ServeHTTP(0xc00002d110, 0xb5a660, 0xc0044141c0, 0xc005018f00) net/http/server.go:2012 +0x44 net/http.serverHandler.ServeHTTP(0xc004414000, 0xb5a660, 0xc0044141c0, 0xc005018f00) net/http/server.go:2807 +0xa3 net/http.(conn).serve(0xc00ef22000, 0xb5bf60, 0xc010532080) net/http/server.go:1895 +0x86c created by net/http.(Server).Serve net/http/server.go:2933 +0x35c	2020-04-02 21:09:55 +03:00
Aliaksandr Valialkin	bf43ad1d4f	docs/Articles.md: move Percona article to third-party	2020-04-02 15:43:19 +03:00
Aliaksandr Valialkin	b6ff251884	docs/Articles.md: add a link to https://blog.cloudera.com/benchmarking-time-series-workloads-on-apache-kudu-using-tsbs/	2020-04-02 15:41:30 +03:00
Aliaksandr Valialkin	cfea171930	docs/CaseStudies.md: add Adsterra case	2020-04-02 00:49:49 +03:00
Aliaksandr Valialkin	3b744f3c32	app/vmstorage: typo fix	2020-04-01 23:43:09 +03:00
Aliaksandr Valialkin	f838cdc86e	app/vmstorage: add `vm_free_disk_space_bytes` metric for monitoring the remaining disk space at `-storageDataPath`	2020-04-01 23:10:44 +03:00
Aliaksandr Valialkin	8a02e01210	docs/Single-server-VictoriaMetrics.md: re-organize chapters	2020-04-01 22:41:00 +03:00
Aliaksandr Valialkin	84fa146792	lib/httpserver: remove unnecessary `http.HandlerFunc` wrapper in gzipHandler	2020-04-01 18:14:47 +03:00
Aliaksandr Valialkin	b3cb188c59	docs/Single-server-VictoriaMetrics.md: small fixes and updates	2020-04-01 18:11:10 +03:00
Aliaksandr Valialkin	d5180dbe78	docs/Cluster-VictoriaMetrics.md: small fixes and updates	2020-04-01 18:06:01 +03:00
Aliaksandr Valialkin	120d452002	docs/Cluster-VictoriaMetrics.md: swap `production build` and `development build` chapters	2020-04-01 17:48:43 +03:00
Aliaksandr Valialkin	0ad7aaf535	lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init	2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin	c189104be7	lib/promscrape: reduce timestamp jitter when scraping targets This should improve compression for timestamps	2020-04-01 16:13:01 +03:00
Aliaksandr Valialkin	4c56acbafa	lib/storage: remove duplicate data points on 7/8minScrapeInterval interval instead of 1/2minScrapeInterval This should reduce storage usage and should improve deduplication accuracy	2020-04-01 15:47:04 +03:00
Aliaksandr Valialkin	29d5fbfcd8	docs/Single-server-VictoriaMetrics.md: mention that environment vars may be prefixed with `-envflag.prefix`	2020-03-31 22:38:26 +03:00
Aliaksandr Valialkin	5792f7296a	README.md: mention that response cache must be reset after import historical data	2020-03-31 19:34:06 +03:00
Aliaksandr Valialkin	504ea876f2	lib/storage: handle errors returned from `TagFilters.Add` when cloning TagFilters with negative filter	2020-03-31 16:18:34 +03:00
Aliaksandr Valialkin	5270b7a097	app/victoria-metrics/testdata: add a test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 12:51:46 +03:00
Aliaksandr Valialkin	ef714e01c1	lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet	2020-03-31 12:35:15 +03:00
Aliaksandr Valialkin	7e755b4bac	lib/storage: optimize per-day inverted index search for tag filters matching big number of time series - Sort tag filters in the ascending number of matching time series in order to apply the most specific filters first. - Fall back to metricName search for filters matching big number of time series (usually this are negative filters or regexp filters).	2020-03-31 00:53:29 +03:00
Aliaksandr Valialkin	d450249955	lib/storage: properly handle `{label=~"foo\|"}` filters as Prometheus does Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched.	2020-03-30 20:21:47 +03:00
Aliaksandr Valialkin	b47444e69d	lib/envflag: add `-envflag.prefix` for setting optional prefix for environment vars	2020-03-30 15:51:44 +03:00
Aliaksandr Valialkin	e6e321f542	vendor: `make vendor-update`	2020-03-30 15:08:00 +03:00
Aliaksandr Valialkin	f4c3a71139	go.mod: update the minimum required Go version from go1.12 to go1.13	2020-03-30 14:57:59 +03:00
Aliaksandr Valialkin	c6cbc0bd19	app/vmselect/prometheus: allow passing relative time to `start`, `end` and `time` args of `/api/v1/*` queries	2020-03-29 21:56:52 +03:00
Aliaksandr Valialkin	cb8696699a	app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()	2020-03-29 21:50:35 +03:00
kreedom	f058efb3d1	[vmalert] config parser (#393 ) * [vmalert] config parser * make linter be happy * fix test * fix sprintf add test for rule validation	2020-03-29 01:49:40 +02:00
Aliaksandr Valialkin	c66a13bf0f	docs: add robots.txt	2020-03-28 23:23:20 +02:00
Aliaksandr Valialkin	ceb6d1459f	docs/vmagent.md: add `prometheus remote_write proxy` use case	2020-03-28 23:17:41 +02:00
Aliaksandr Valialkin	8d55af4e75	docs/CaseStudies.md: add Brandwatch case study	2020-03-28 20:58:46 +02:00
Aliaksandr Valialkin	253844b74c	deployment/docker: run docker apps under default user (0, root) in order to preserve backwards compatibility If docker app is upgraded from root to non-root, then the data pointed by `-storageDataPath` or similar flags becomes denied to non-root user after the upgrade. This breaks upgrade path. So revert back to default root user for docker apps. Users may explicitly execute `docker run --user <non_root_user>` for running docker apps under non-root user.	2020-03-28 19:23:13 +02:00
Roman Khavronenko	a2767fe86f	bump Prometheus and Grafana images (#389 )	2020-03-28 01:19:19 +02:00
Roman Khavronenko	9373a62f8a	Update dashboard according to new Grafana version and some metric renames. (#392 ) The list of changes is following: * fix Uptime panel column styles according to changes introduced in 6.7 Grafana version * fix panel `vminsert/Rows per insert` due to metric rename - see #336 * change default datasource to VictoriaMetrics since dashboard now uses MetricsQL for `vminsert/Rows per insert` panel	2020-03-28 01:17:38 +02:00
Dmitry Naumov	b84071fc25	Rootless docker images by default (#358 ) * Rootless docker images by default * Migrate to rootless base image Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-03-27 21:18:32 +02:00
Aliaksandr Valialkin	b803bcca6b	vendor: `make vendor-update`	2020-03-27 20:17:26 +02:00
Aliaksandr Valialkin	42c290ce9f	lib/httpserver: add `-http.maxGracefulShutdownDuration` command-line flag for tuning the maximum duration required for graceful shutdown of http server	2020-03-27 20:10:05 +02:00
Aliaksandr Valialkin	8fa80a2dbc	lib/uint64set: remove zero buckets after Set.Intersect	2020-03-27 01:16:34 +02:00
Aliaksandr Valialkin	7a35447031	lib/uint64set: small code cleanup and perf tuning * Remember the last accessed bucket on Has() call. * Inline fast paths inside Add() and Has() calls. * Remove fragile code with maxUnsortedBuckets inside bucket32.	2020-03-25 15:29:59 +02:00
Aliaksandr Valialkin	19d93e1a2e	deployment/docker: update Go builder from Go1.14.0 to Go1.14.1	2020-03-24 22:35:46 +02:00
Aliaksandr Valialkin	cce936de5b	lib/uint64set: go fmt	2020-03-24 22:28:09 +02:00
Aliaksandr Valialkin	7cdac6634c	lib/storage: serialize snapshot creation process with mutex This guarantees that the snapshot contains all the recently added data from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.	2020-03-24 22:27:28 +02:00
Aliaksandr Valialkin	c31b956355	lib/uint64set: added more tests	2020-03-24 22:27:26 +02:00
Aliaksandr Valialkin	8fa9066b98	docs/CaseStudies.md: added a case study from MHI Vestas Offshore Wind	2020-03-14 13:22:41 +02:00
Aliaksandr Valialkin	31a533656e	lib/storage: remove obsolete code	2020-03-13 22:42:42 +02:00
Aliaksandr Valialkin	58cb7fc476	app/vmselect: adjust `label_map()` handling for corner cases The following corner cases now supported: * label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label` * label_map(q, "label", "foo", "") - removes `label="foo"` from series All the unmatched labels are kept unchanged.	2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin	5c5a30734e	vendor: update github.com/VictoriaMetrics/metrics from v1.11.0 to v1.11.2 This fixes data race in Histogram	2020-03-13 12:40:15 +02:00
Aliaksandr Valialkin	bf1869d33d	lib/promscrape: allow overriding external_labels as Prometheus does Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say: > In communication with external systems, they are always applied only > when a time series does not have a given label yet and are ignored otherwise. Though this may result in consistency chaos when scrape targets override `external_labels`, let's stick with Prometheus behavior for the sake of backwards compatibility. There is last resort in vmagent with `-remoteWrite.label`, which consistently sets the configured labels to all the metrics before sending them to remote storage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366	2020-03-12 20:25:27 +02:00
Aliaksandr Valialkin	0e7a71a245	app/vmselect: add `label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN)` function to MetricsQL Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin	39a977b5aa	vendor: update google.golang.org/genproto from fc8f55426688 to da6875a35672	2020-03-12 18:12:46 +02:00
Aliaksandr Valialkin	9ef7bba17b	vendor: update golang.org/x/tools from 26f6a1b6802d to 5e2df02acb1e	2020-03-12 18:08:02 +02:00
Aliaksandr Valialkin	d91790543f	vendor: update github.com/aws/aws-sdk-go from v1.29.10 to v1.29.22	2020-03-12 17:55:11 +02:00
Aliaksandr Valialkin	fa4d70b428	vendor: update google.golang.org/api from v0.19.0 to v0.20.0	2020-03-12 17:52:02 +02:00
Aliaksandr Valialkin	83fe650ca1	vendor: update golang.org/x/sys from d5e6a3e2c0ae to 5c8b2ff67527	2020-03-12 17:46:39 +02:00
Aliaksandr Valialkin	e5c073a9a1	vendor: update github.com/klauspost/compress from v1.10.1 to v1.10.3	2020-03-12 17:33:11 +02:00
Aliaksandr Valialkin	fa7910fba1	lib/protoparser/csvimport: add missing metric `vm_rows_invalid_total{type="csvimport"}`	2020-03-12 15:28:10 +02:00
Aliaksandr Valialkin	41b532046f	README.md: mention about alternative dashboard for cluster version - https://grafana.com/grafana/dashboards/11831	2020-03-12 15:11:55 +02:00
Aliaksandr Valialkin	50555d89d3	app/vmselect: add `-search.maxStalenessInterval` for tuning Prometheus data model closer to Influx-style data model	2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin	375d5483fa	lib/promscrape: remove possible races when registering and de-registering scrape workers for `/targets` page	2020-03-11 16:30:43 +02:00
Aliaksandr Valialkin	b46af9678e	app/vmagent: mention that vmagent can filter data	2020-03-11 16:23:10 +02:00
Aliaksandr Valialkin	803f919c75	docs/Articles.md: add a link to https://stas.starikevich.com/posts/disk-usage-for-vm-versus-prometheus/	2020-03-11 04:56:37 +02:00
Aliaksandr Valialkin	187fd89c70	lib/promscrape: consistently update `/targets` page after SIGHUP	2020-03-11 03:20:38 +02:00
Aliaksandr Valialkin	8939c19281	app/vmstorage: return 500 status code instead of 200 status code on internal errors inside `/snapshot/*` handlers	2020-03-10 23:54:27 +02:00
Aliaksandr Valialkin	b51e548b64	docs/vmagent.md: sync with app/vmagent/README.md	2020-03-10 21:54:35 +02:00
Aliaksandr Valialkin	f6410ff2bf	app/vmselect: add optional `max_rows_per_line` query arg to `/api/v1/export` This arg allows limiting the number of data points that may be exported on a single line.	2020-03-10 21:47:43 +02:00
Aliaksandr Valialkin	2f0a36044c	app/{vmagent,vminsert}: add support for importing csv data via `/api/v1/import/csv`	2020-03-10 21:17:40 +02:00
Aliaksandr Valialkin	7545784a49	all: fix `golangci-lint` issues	2020-03-10 19:40:03 +02:00
Aliaksandr Valialkin	8a2ea0171a	docs/FAQ.md: actualize answer about deduplication	2020-03-09 13:37:39 +02:00
Aliaksandr Valialkin	d70c9b9556	docs: add missing vmagent.png, which is used in vmagent.md	2020-03-09 13:37:38 +02:00
Aliaksandr Valialkin	3fc6599aa2	app/vmagent: properly apply `-remoteWrite.sendTimeout` to fasthttp.HostClient	2020-03-09 13:31:22 +02:00
Aliaksandr Valialkin	d39dd8aa69	lib/promscrape: do not retry idempotent requests when scraping targets This should prevent from the following unexpected side-effects of idempotent request retries: - increased actual timeout when scraping the target comparing to the configured scrape_timeout - increased load on the target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357	2020-03-09 13:31:20 +02:00
Aliaksandr Valialkin	12789a4621	app/vmagent: do not allow non-supported fields in `-remoteWrite.relabelConfig` and `file_sd_configs` This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363	2020-03-06 20:19:34 +02:00
Aliaksandr Valialkin	47e986c26f	app/vmagent: properly add labels set via `-remoteWrite.label` to metrics before sending them to `-remoteWrite.url`	2020-03-06 19:28:14 +02:00
Aliaksandr Valialkin	0d893eff36	Makefile: add build and test rules with enabled race detector. These rules have `-race` suffix Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin	197d2916ab	README.md: add a link to https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Articles	2020-03-04 20:06:22 +02:00
Aliaksandr Valialkin	7909964cf3	docs/Articles.md: add a link to https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/	2020-03-04 20:06:21 +02:00
Aliaksandr Valialkin	0176fc4206	app/vmagent/README.md: small fixes	2020-03-04 18:15:24 +02:00
Aliaksandr Valialkin	ac03be5a2c	app/vmagent/README.md: typo fix	2020-03-04 18:05:43 +02:00
Aliaksandr Valialkin	9354b9177a	app/vmagent/README.md: clarification	2020-03-04 18:04:06 +02:00
Aliaksandr Valialkin	4302555228	app/vmagent/README.md: add `iot and edge monitoring` use case	2020-03-04 18:01:40 +02:00
Aliaksandr Valialkin	ea5904fd76	app/vmagent/README.md: add `use cases` section	2020-03-04 17:42:57 +02:00
Aliaksandr Valialkin	ed355fe6b4	docs/CaseStudies: add Synthesio	2020-03-04 17:12:18 +02:00
Aliaksandr Valialkin	50190263c8	docs/Single-server-VictoriaMetrics.md: sync with README.md	2020-03-03 21:39:31 +02:00
Aliaksandr Valialkin	31a76a7b3a	lib/promscrape: consistency renaming: stopCh -> globalStopCh	2020-03-03 20:08:22 +02:00
Aliaksandr Valialkin	f01d1bf4a8	app/vmagent: add `-remoteWrite.maxDiskUsagePerURL` for limiting the maximum disk usage for each `-remoteWrite.url` buffer Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352	2020-03-03 19:49:20 +02:00
Aliaksandr Valialkin	808c17e250	app/vmagent/remotewrite: do not reset empty relabelCtx	2020-03-03 15:01:21 +02:00
Aliaksandr Valialkin	af19ca2483	app/vmagent: add `-remoteWrite.urlRelabelConfig` for applying individual relabeling for each `-remoteWrite.url` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308	2020-03-03 13:13:06 +02:00
Aliaksandr Valialkin	c3b239eb1a	lib/protoparser/prometheus: allow trailing comma in tags list The trailing comma is generated by cloudwatch exporter. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/350	2020-03-02 22:23:28 +02:00
Aliaksandr Valialkin	d23df53ba2	app/vmselect/prometheus: do not add `__name__!=` filter when searching for all the matching metric names via `/api/v1/label/__name__/values` with non-empty label filter This should reduce query time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 23:36:38 +02:00
Aliaksandr Valialkin	6a1aab88fd	lib/protoparser: metrics renaming: `vm_protoparser_<type>_` -> `vm_protoparser_{type="<type>"}` This should improve composability of these metrics in PromQL queries	2020-02-28 20:19:59 +02:00
Aliaksandr Valialkin	0eed71c7f4	app/vmagent/remotewrite: yet another typo fix	2020-02-28 20:07:00 +02:00
Aliaksandr Valialkin	c2e602286c	lib/persistentqueue: reset chunk file when the persistent queue is empty	2020-02-28 20:06:59 +02:00
Aliaksandr Valialkin	6cdc97a53f	app/vmagent/remotewrite: typo fix	2020-02-28 19:05:11 +02:00
Aliaksandr Valialkin	cc39c9d74b	app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage	2020-02-28 18:58:13 +02:00
Aliaksandr Valialkin	6282b29a44	docs/Single-server-VictoriaMetrics.md: sync with README.md	2020-02-28 12:59:02 +02:00
Aliaksandr Valialkin	45d21d18a8	docs: add a doc for vmagent	2020-02-28 12:23:44 +02:00
Aliaksandr Valialkin	8fa1cd24d8	app/vmselect/prometheus: properly pass filter for `labelName=__name__` in labelValuesWithMatches Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 12:17:30 +02:00
Aliaksandr Valialkin	cf9aee4ec3	all: properly split `vm_deduplicated_samples_total` among cluster components Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin	5e7b4795bd	lib/envflag: typo fix in docs to `-envflag.enable`: envoronment->environment	2020-02-27 21:56:46 +02:00
Aliaksandr Valialkin	52fe4e68fb	deployment/docker: update Go builder from Go1.13.8 to Go1.14.0	2020-02-26 22:14:43 +02:00
Aliaksandr Valialkin	1286cead75	app/vminsert: properly initialize InsertCtx This should prevent from panic described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/339	2020-02-26 21:21:02 +02:00
Aliaksandr Valialkin	0597f1e39a	app/vmagent: allow setting `-httpListenAddr` to empty string in order to disable listening for http requests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340	2020-02-26 20:58:26 +02:00
Aliaksandr Valialkin	8c2d396e8a	`make vendor-update`	2020-02-26 20:46:24 +02:00
Aliaksandr Valialkin	a6c0d490a3	vendor: update github.com/VictoriaMetrics/metrics from v1.10.1 to v1.11.0	2020-02-26 20:40:34 +02:00
Aliaksandr Valialkin	266101feb4	app/vmagent/README.md: list service discovery mechanisms, which will be added soon Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-02-26 19:28:19 +02:00
Aliaksandr Valialkin	e6a481ab11	lib/promscrape: properly reload new configs on SIGHUP	2020-02-26 13:54:24 +02:00
Aliaksandr Valialkin	fa6815712f	lib/promscrape: go fmt	2020-02-26 13:24:40 +02:00
Edouard Hur	fed37ecfcb	add envvars details (#337 )	2020-02-26 13:23:06 +02:00
Aliaksandr Valialkin	f2a6948a14	lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331	2020-02-25 20:50:18 +02:00
Aliaksandr Valialkin	c6c7843e93	app/vmagent: add `-remoteWrite.maxBlockSize` command-line flag for limiting the maximum size of unpacked block to send to remote storage	2020-02-25 19:58:11 +02:00
Aliaksandr Valialkin	c4194020ef	app/vmagent: do not allow sending unpacked requests with sizes exceeding `-maxInsertRequestSize`	2020-02-25 19:35:43 +02:00
Aliaksandr Valialkin	2471340e0d	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:18:01 +02:00
Aliaksandr Valialkin	f96fb93ca5	app/vmagent: add a counter for `/targets` handler calls	2020-02-25 18:17:25 +02:00
Aliaksandr Valialkin	25c570dae7	app/vmagent/README.md: mention that `vmagent` exposes target statuses at `/targets` page	2020-02-25 18:16:08 +02:00
Aliaksandr Valialkin	7a045125cc	lib/fs: typo fix: read blocks bigger than 8KB via `pread()` call instead of using mmap	2020-02-25 18:04:06 +02:00
Aliaksandr Valialkin	ca28a3e805	app/vmagent: logo fix	2020-02-25 00:09:55 +02:00
Aliaksandr Valialkin	777a39f7a1	app/vmagent: update docs	2020-02-25 00:09:53 +02:00
Aliaksandr Valialkin	61e67b8922	app/vmagent/README.md: small fixes	2020-02-24 21:26:12 +02:00
Aliaksandr Valialkin	13ee8271d0	lib/envflag: substitute dots with underscores in env var names if -envflag.enable is set See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-24 21:15:11 +02:00
Aliaksandr Valialkin	6ca1e58d98	app/vmselect/promql: properly take into account the first datapoint when calculating `rollup_candlestick`	2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin	b58e3fc8a9	app/vmselect/promql: do not take into account values outside the current window in `rollup_candlestick` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-23 18:06:26 +02:00
Yaroslav	c69d4b01f0	fix rollupOpen(), rollupHigh(), rollupLow() functions (#328 )	2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin	7ee7614e90	app/vmagent: initial implementation for vmagent	2020-02-23 17:31:54 +02:00
Aliaksandr Valialkin	ab1e66d31f	vendor: update github.com/valyala/fastjson from v1.4.5 to v1.5.0	2020-02-23 10:06:48 +02:00
Aliaksandr Valialkin	f22aefdb16	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin	110cce24d9	lib/storage: add vm_ prefix to `deduplicated_samples_total` metric	2020-02-21 19:33:36 +02:00
Aliaksandr Valialkin	d5c2a0ce64	app/vmselect: add `-search.cacheTimestampOffset` command-line flag This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin	c70822db50	app/vmselect: add `/internl/resetRollupResultCache` handler for resetting response cache Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 14:02:12 +02:00
Aliaksandr Valialkin	51abc84932	deployment/docker: update Go builder from v1.13.7 to v1.13.8	2020-02-20 19:56:36 +02:00
Aliaksandr Valialkin	9d279e26a7	lib/protoparser/prometheus: skip leading whitespace from tag names	2020-02-16 19:06:23 +02:00
Aliaksandr Valialkin	fb5848f536	Makefile: add missing `vmbackup` and `vmrestore` to `all` and `all-pure` targets	2020-02-16 16:55:34 +02:00
Aliaksandr Valialkin	d687e5518d	vendor: `make vendor-udpate`	2020-02-16 16:12:28 +02:00
Aliaksandr Valialkin	a2b81b71b9	lib/storage: typo fix	2020-02-16 15:53:48 +02:00
Aliaksandr Valialkin	ad4cb9f3ca	lib/storage: prevent from clobbering nin-nil lastError in Storage.add	2020-02-16 15:51:35 +02:00
Aliaksandr Valialkin	afecb34491	app/vmstorage: limit the maximum error message size before sending it to client Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/315	2020-02-13 17:33:12 +02:00
Aliaksandr Valialkin	846d7fa7e9	app/vmselect: add `sort_by_label(q, label)` and `sort_by_label_desc(q, label)` functions This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin	e3b18ca1ab	lib/mergeset: skip createing temporary part objects when merging source inmemory parts This should reduce CPU usage when adding new entries to inverted index. This should alos prevent from creating stalled cleaner goroutines for the created temporary parts, since they were never closed. This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 14:09:13 +02:00
Aliaksandr Valialkin	347aaba79d	lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin	6e0013ca39	app/vmselect/prometheus: typo fix in `-latencyOffset` description	2020-02-12 14:00:38 +02:00
Aliaksandr Valialkin	22ede83146	`make vendor-update`	2020-02-10 23:35:14 +02:00
Aliaksandr Valialkin	ebf7785d79	vendor: update github.com/VictoriaMetrics/metrics from v1.9.3 to v1.10.1	2020-02-10 23:07:29 +02:00
Aliaksandr Valialkin	e7d1037210	docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent	2020-02-10 23:03:31 +02:00
Edouard Hur	e8f92a4ee8	Cluster - prometheus metrics fix (#314 ) * add missing '/{}' in prom query range requests * fix missing leading '/' on prom lavelValuesErrors path	2020-02-10 22:15:21 +02:00
Aliaksandr Valialkin	fcdd95a6ef	lib/envflag: check for incorrect flag values read from environment vars	2020-02-10 16:09:03 +02:00
Aliaksandr Valialkin	9c5db9400c	lib/envflag: add `-envflag.enable` command-line flag for enabling reading flags from environment vars By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 16:09:01 +02:00
Aliaksandr Valialkin	1010a57882	all: allow setting flags via environment vars Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin	ea66212c93	lib/storage: move `-dedup.minScrapeInterval` flag outside lib/storage, so it doesnt show up in `vminsert` in cluster version	2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin	07c067697e	docs/Single-server-VictoriaMetrics.md: sync with README.md	2020-02-07 00:03:05 +02:00
Aliaksandr Valialkin	e6d9ea3094	app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit `9e1119dab8`	2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin	4a1de7fee9	app/vmselect/promql: properly adjust time range for data to select Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin	8e77b54846	app/vmselect: unconditionally `offset -step` to `rollup_candlestick`. This makes results more consistent	2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin	ce38b176bc	app/vmselect/promql: automatically apply `offset -step` to `rollup_candlestick` function in order to obtain the expected OHLC results See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin	4f7116d1ee	app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin	8b360a25e9	lib/logger: initialize output to os.Stderr by default	2020-02-04 22:43:26 +02:00
Aliaksandr Valialkin	c931a540f4	Do not require checking for errors returned from `fmt.Fprint` This fixes `make errcheck` error found in lib/logger	2020-02-04 22:03:52 +02:00
Aliaksandr Valialkin	1f271a9815	lib/logger: add `-loggerOutput` command-line flag This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306	2020-02-04 21:48:24 +02:00
Aliaksandr Valialkin	49ab3fa076	lib/logger: do not clutter `-loggerFormat=json` output with stack trace This should improve json parsing	2020-02-04 21:40:20 +02:00
Aliaksandr Valialkin	56d6b8ed0a	lib/storage: do not deduplicate blocks with less than 32 samples during merge This should improve deduplication accuracy for blocks with higher number of samples.	2020-02-04 18:41:37 +02:00
Aliaksandr Valialkin	ccd3aa4f15	app/vmselect: take into account the time the requests wait in the queue if `-search.maxConcurrentRequests` is exceeded This will prevent from excess CPU usage for timed out queries.	2020-02-04 16:20:48 +02:00
Aliaksandr Valialkin	e6bf88a4d4	app/vmselect: add a placeholder for `/api/v1/metadata`, which could be requested by Grafana See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.	2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin	7cde594696	all: do not clash flag description with back-quoted flag types See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:56:01 +02:00
Edouard Hur	2ec248453b	do not fill max lines (#307 )	2020-02-03 21:21:04 +00:00
Roman Khavronenko	ce8eb8a207	improve description for `Pending datapoints` panel; (#301 ) Use bits/s for network usage panels;	2020-02-03 02:07:07 +02:00
Aliaksandr Valialkin	45bc6c62f2	app/vmselect/promql: adjust `and` and `unless` binary operator handling to be consistent with Prometheus	2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin	36ea1b503b	deployment/docker: update Go builder from v1.13.6 to v1.13.7	2020-01-31 18:06:10 +02:00
Aliaksandr Valialkin	9b25a2fb67	lib/fs: remove unused `readerAt` interface	2020-01-31 15:13:00 +02:00
Aliaksandr Valialkin	e3adc095bd	all: add `-dedup.minScrapeInterval` command-line flag for data de-duplication Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin	a45f25699c	lib/storage: re-use indexSearch inside Storage.prefetchMetricNames	2020-01-31 01:18:53 +02:00
Aliaksandr Valialkin	cb5c39ee70	lib/fs: optimize small reads for `ReaderAt.MustReadAt` by reading from memory-mapped space instead of reading from file descriptor This should improve performance when reading many small blocks.	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	da19fffa08	all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	1332ddc15e	lib/storage: pass missing AccountID and ProjectID to searchMetricName	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	4ed5e9a7ce	lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init This should speed up Search.NextMetricBlock loop for big number of found time series.	2020-01-30 15:16:16 +02:00
Alexander Danilov	ced989c966	Fix current/max graphs (#298 )	2020-01-29 23:48:36 +00:00
Aliaksandr Valialkin	cb2a2f281f	lib/mergeset: properly update `lastAccesstime` in indexBlockCache entries This is a follow-up for `6665f10e7b`	2020-01-29 21:21:01 +02:00
Aliaksandr Valialkin	170c1c3a4e	app/vmselect/promql: add `keep_next_value(q)` for filling gaps with the next non-empty value	2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin	b3bd64fdb2	docs/Single-server-VictoriaMetrics.md: fix heading size for `Third-party contributions` section	2020-01-28 23:14:06 +02:00
Aliaksandr Valialkin	a9c1d5b351	app/vminsert: moved `-maxInsertRequestSize` command-line flag out of `lib/prompb` in order to prevent its inclusion in `vmselect` and `vmstorage` apps	2020-01-28 22:53:50 +02:00
Aliaksandr Valialkin	b28c9a3944	app/vmselect/promql: return expected results from `increase()` over the beginning of time series, which start from big value Examples for such counters: OS-level counters for network or cpu stats.	2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin	11c03328ae	app/victoria-metrics: remove `integration` build tag from tests This simplifies testing with `go test ./app/victoria-metrics` without the need to remember to pass `-tags=integration` to Go commands.	2020-01-27 20:27:39 +02:00
Aliaksandr Valialkin	9a02ca67e9	docs/Single-server-VictoriaMetrics.md: sync with master	2020-01-27 18:47:46 +02:00
Aliaksandr Valialkin	dab9a63485	docs/Single-server-VictoriaMetrics.md: update `Retention` section	2020-01-27 18:46:32 +02:00
Aliaksandr Valialkin	2bb9b089d5	README.md: mention https://github.com/AnchorFree/tsdb-remote-write	2020-01-27 18:36:22 +02:00
Aliaksandr Valialkin	3e304890a6	app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296	2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin	81ba371eaf	lib/logger: fix improperly set skipframes for all the logging functions	2020-01-26 18:34:58 +02:00
Aliaksandr Valialkin	9f595cb2b1	lib/httpserver: log the caller of httpserver.Errorf Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.	2020-01-25 20:18:06 +02:00
Aliaksandr Valialkin	4d70a81e18	app/vminsert: do not drop pending rows if all the vmstorage backends are unavailable Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/294	2020-01-24 22:10:10 +02:00
Aliaksandr Valialkin	36a1a21d6e	lib/protoparser: add parser for Prometheus exposition text format This parser will be used by vmagent	2020-01-24 20:11:19 +02:00
Aliaksandr Valialkin	0cda6afa8e	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:55:18 +02:00
Aliaksandr Valialkin	a9802fcb72	docs/Articles.md: add a link to https://medium.com/@valyala/billy-how-victoriametrics-deals-with-more-than-500-billion-rows-e82ff8f725da	2020-01-22 19:08:45 +02:00
Aliaksandr Valialkin	ea53a21b02	all: consistently log durations in seconds with millisecond precision This should improve logs readability	2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin	6eddce1d15	vendor: `make vendor-update`	2020-01-22 18:09:24 +02:00
Aliaksandr Valialkin	e1a264173a	app/vmselect: mention the original query and time range in error messages This should simplify debugging invalid or heavy queries.	2020-01-22 17:34:35 +02:00
Aliaksandr Valialkin	18a4503261	vendor: update github.com/klauspost/compress from v1.9.7 to v1.9.8 New version should have better gzip compression. See https://github.com/klauspost/compress#changelog	2020-01-22 16:51:17 +02:00
Aliaksandr Valialkin	3c6ae8c947	docs: Mention Slack and Telegram channels for user questions	2020-01-22 16:51:16 +02:00
Aliaksandr Valialkin	e127173984	app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors	2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin	f3b9f8b823	app/vmselect/prometheus: increase default value `-maxExportDuration` to 30 days, since 10 minutes beat users exporting bit amounts of data	2020-01-22 15:53:41 +02:00
Aliaksandr Valialkin	be5adbfda4	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.5 to v1.5.7	2020-01-22 12:30:55 +02:00
Aliaksandr Valialkin	40e564eb9c	app/vmselect/promql: add `range_over_time(m[d])` function for calculating value range for `m` over `d`	2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin	ecddba30fe	app/vminsert/netstorage: increase timeout for pushing data from vminsert to vmstorage by 3x Our clients report that the previous timeout could lead to frequent errors when vmstorage starts background merge for big parts on slow HDD.	2020-01-21 18:21:49 +02:00
Aliaksandr Valialkin	9eaa2ab871	app/vmselect/promql: add `label_match(q, label, regexp)` and `label_mismatch(q, label, regexp)` functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin	62b041e90a	lib/{mergeset,storage}: properly update `lastAccessTime` in index and data block cache entries	2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin	b297fec515	README.md: mention that `delete API` shouldnt be used on a regular basis due to non-zero overhead	2020-01-20 13:28:58 +02:00
Aliaksandr Valialkin	d3b4b0f492	docs/FAQ.md: typo fix according to comment from https://www.reddit.com/message/messages/lezkmo	2020-01-18 18:05:22 +02:00
Aliaksandr Valialkin	179c7db4c9	docs/CaseStudies.md: add links to COLOPL talk about VictoriaMetrics	2020-01-18 17:24:00 +02:00
Aliaksandr Valialkin	cbd0452317	app/vminsert: increase default value for `-insert.maxQueueDuration` from 30s to 60s This should help catching up with high ingestion rate after VictoriaMetrics restart.	2020-01-18 14:39:30 +02:00
Aliaksandr Valialkin	607d4418b8	lib/uint64set: add missing bucket32.b16his values	2020-01-18 14:26:23 +02:00
Aliaksandr Valialkin	e3379537cd	lib/uint64set: optimize Set.Union This should improve performance for queries over big number of time series	2020-01-18 13:47:34 +02:00
Aliaksandr Valialkin	5077efd3f7	lib/uint64set: add benchmarks for Set.Union	2020-01-18 13:47:33 +02:00
Aliaksandr Valialkin	a851c75703	lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64)	2020-01-17 23:59:19 +02:00
Aliaksandr Valialkin	2084921e64	all: use `github.com/klauspost/compress/gzip` instead of `compress/gzip` `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:59:17 +02:00
Aliaksandr Valialkin	ab4d5d72eb	lib/uint64set: reduce memory allocations in Set.AppendTo	2020-01-17 22:33:00 +02:00
Aliaksandr Valialkin	476c7fb109	lib/storage: reduce memory allocations when merging metricID sets	2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin	29d21259f0	lib/uint64set: typo fix in Set.Intersect	2020-01-17 18:11:46 +02:00
Aliaksandr Valialkin	54db08a60f	app/vmselect/netstorage: `make fmt`	2020-01-17 17:46:20 +02:00
Aliaksandr Valialkin	d21cc2d16a	app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file This should reduce memory usage on systems with more than 8GB RAM.	2020-01-17 16:28:28 +02:00
Aliaksandr Valialkin	ed1d259b10	lib/uint64set: optimize Intersect, Subtract and Union functions This should improve performance for queries over big number of time series.	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	68d35357b1	lib/uint64set: improve benchmark for Set.Intersect	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	b05f6cf11c	app/vmselect: limit the default value for `-search.maxConcurrentRequests`, so it plays well on systems with more than 16 vCPUs A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value. This will give more chances for executing insert path.	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	a9f683423c	app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	ffe352ad31	lib/uint64set: add benchmark for Set.Intersect	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	bdfb219992	`make vendor-update`	2020-01-16 14:17:48 +02:00
Aliaksandr Valialkin	4b16b7fd11	all: mention command-line flags used for limiting the incoming request size in error messages This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin	ce0b602405	app/vmselect/promql: fix panic on `sum(aggr_over_time(...))` with incorrect number of args	2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin	7d429e2806	lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods Iterate items with newly added Set.ForEach method instead of allocating `[]uint64` slice for all the items before the iteration.	2020-01-15 12:15:48 +02:00
Aliaksandr Valialkin	4ecb7f15b6	docs/FAQ.md: add bullet comparison with Cortex and Thanos	2020-01-15 10:47:46 +02:00
Aliaksandr Valialkin	caffb0cd01	lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)	2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin	b03ccbf6f7	lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches This should reduce memory usage in the long run when old blocks and indexes aren't accessed anymore.	2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin	8a4d4978a3	deployment/docker: update Prometheus from v2.14.0 to v2.15.2 and Grafana from v6.5.0 to v6.5.2	2020-01-12 23:14:56 +02:00
Aliaksandr Valialkin	cbafb7ae59	docs: add a link to VictoriaMetrics subreddit - https://www.reddit.com/r/VictoriaMetrics/	2020-01-12 00:07:06 +02:00
Aliaksandr Valialkin	bcd3f0c5bd	app/vmselect/promql: add `hoeffding_bound_upper(phi, m[d])` and `hoeffding_bound_lower(phi, m[d])` functions These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin	fc01b11ddc	app/vmselect/promql: return continuous values for `min_over_time` and `max_over_time` when `step` is smaller than `scrape_interval`	2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin	92e00779fa	deployment/docker: switch Go builder from v1.13.5 to v1.13.6	2020-01-11 11:06:12 +02:00
Aliaksandr Valialkin	2cacea8c64	README.md: mention about Prometheus->VictoriaMetrics exporter https://github.com/ryotarai/prometheus-tsdb-dump	2020-01-11 01:28:50 +02:00
Aliaksandr Valialkin	16fb128bbc	app/vmselect/promql: do not take into account the previous point before time window in square brackets for `min_over_time`, `max_over_time`, `rollup_first` and `rollup_last` functions This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.	2020-01-11 00:26:11 +02:00
Aliaksandr Valialkin	1c445bf7eb	vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5 This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.	2020-01-10 23:14:34 +02:00
Aliaksandr Valialkin	adc36d00b7	app/vmselect/promql: properly handle `aggr(aggr_over_time(...))`	2020-01-10 21:57:11 +02:00
Aliaksandr Valialkin	87a106702b	app/vmselect/promql: add `aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])` function This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin	c314d9a219	app/vmselect/promql: add `tmin_over_time(m[d])` and `tmax_over_time(m[d])` functions These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin	706b33dc82	docs: fix spelling typos	2020-01-09 23:43:45 +02:00
Aliaksandr Valialkin	1029b6ab34	lib/backup/s3remote: check whether the file exists before deleting it Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284	2020-01-09 23:20:51 +02:00
Aliaksandr Valialkin	705af61587	app/{vmbackup,vmrestore}: add `backup complete` file to backup when it is complete and check for this file before restoring from backup This should prevent from restoring from incomplete backups. Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.	2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin	7edbd930d5	vendor: update github.com/valyala/fastjson from v1.4.1 to v1.4.2 This fixes parsing of `inf` and `nan` values in json lines passed to `/api/v1/import`	2020-01-08 20:48:08 +02:00
Aliaksandr Valialkin	da62e894dd	README.md: remove `height="200px"` from logo image, since it is improperly displayed on smartphones	2020-01-08 20:29:37 +02:00
Aliaksandr Valialkin	24a852f900	README.md: typo fix	2020-01-08 14:44:28 +02:00
Aliaksandr Valialkin	7c6df1e51d	app/vmselect/promql: skip `rate` calculation for the first point on time series	2020-01-08 14:42:44 +02:00
Aliaksandr Valialkin	7d8d921db9	docs: add references to `Remote Write Storage Wars` Also mention than VictoriaMetrics uses less RAM than Thanos Store Gateway - see https://github.com/thanos-io/thanos/issues/448 for details.	2020-01-04 23:58:27 +02:00
Aliaksandr Valialkin	53e176ed67	lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition This should reduce write amplification for high ingestion rate on multi-CPU systems	2020-01-04 23:58:23 +02:00
Aliaksandr Valialkin	1f941875db	docs/CaseStudies.md: add link to `Remote Write Storage Wars` talk from Adidas at PromCon 2019	2020-01-04 16:51:02 +02:00
Aliaksandr Valialkin	76707b2ab9	app/vmselect/promql: fix calculations for `histogram_share`	2020-01-04 14:44:15 +02:00
Aliaksandr Valialkin	89b551201c	lib/metricsql: export IsRollupFunc and IsTransformFunc, since they can be used by package users	2020-01-04 13:25:13 +02:00
Aliaksandr Valialkin	8cfd4decea	LICENSE: update year	2020-01-04 13:21:11 +02:00
Aliaksandr Valialkin	accad01b3e	app/vmselect/promql: add missing MetricName into netstorage.Result in tests	2020-01-04 12:53:14 +02:00
Aliaksandr Valialkin	6f29d37cb5	app/vmselect/promql: add `histogram_share(le, buckets)` function	2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin	2290503140	app/vmselect/promql: add `absent_over_time(m[d])` func similar to the function in Prometheus 2.16 See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin	67f94bbe12	app/vmselect/promql: add `histogram_over_time(m[d])` rollup function	2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin	9a1f6848ca	app/vmselect/promql: fix results caching for multi-arg rollup functions such as `quantile_over_time` Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.	2020-01-03 20:44:54 +02:00
Aliaksandr Valialkin	3d0c7b095a	app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating `rate` for the first point on the time series This should provide better estimation for `rate` in the beginning of time series.	2020-01-03 19:03:32 +02:00
Aliaksandr Valialkin	588531dd76	lib/uint64set: reduce memory usage when storing big number of sparse metric_id values	2020-01-03 18:17:17 +02:00
Aliaksandr Valialkin	6ea7f23446	app/vmselect/promql: increase the estimated number of time series returned by `aggr() by (something)` from 100 to 1K, since 100 may result in OOM for high number of time series	2020-01-03 01:02:30 +02:00
Aliaksandr Valialkin	e0abf45d45	app/vmselect/promql: add `share_le_over_time` and `share_gt_over_time` functions for SLI and SLO calculations	2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin	19962e2732	docs: refer to standalone MetricsQL package	2020-01-02 23:43:43 +02:00
Aliaksandr Valialkin	a15a6d9ac1	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.4 to v1.5.5	2019-12-29 18:18:12 +02:00
Aliaksandr Valialkin	0d2e83e9d7	lib/metricsql: add example for ExpandWithExprs	2019-12-26 21:31:15 +02:00
Aliaksandr Valialkin	e3ae813e6a	vendor: `make vendor-update`	2019-12-26 19:42:53 +02:00
Aliaksandr Valialkin	940c55f9d1	vendor: update github.com/valyala/gozstd from v1.6.3 to v1.6.4 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-26 19:31:07 +02:00
Aliaksandr Valialkin	eb1a66c577	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin	453d71d082	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:09:09 +02:00
Mike Poindexter	009d1559db	Split Extended PromQL parsing to a separate library	2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin	ff18101d30	app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range This should prevent from the following panic at app/vmselect/promql/binary_op.go:255: BUG: len(leftVaues) must match len(rightValues) and len(dstValues)	2019-12-24 22:45:49 +02:00
Aliaksandr Valialkin	f22c9dbb0f	lib/fs: typo fix in fadvise_unix.go	2019-12-24 21:00:04 +02:00
Aliaksandr Valialkin	d3c185f0ca	lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin	091e35cf0c	lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin	0e51058a0d	lib/encoding: mention unpacked block size in the error message if unparsed tail left Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin	e24ee43109	app/vmselect/promql: adjust calculations for `rate` and `increase` for the first value These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.	2019-12-24 19:41:03 +02:00
Aliaksandr Valialkin	9a2554691c	app/vmselect/promql: properly calculate `rate` on the first data point It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape, i.e. we can assume its value was 0 at this time.	2019-12-24 15:55:15 +02:00
Aliaksandr Valialkin	97de50dd4c	app/vmselect/netstorage: improve error message when reading data size in readBytes	2019-12-24 14:40:14 +02:00
Aliaksandr Valialkin	c0060c5858	deployment/docker: remove Docker image tag in docker-compose images This should allow loading latest images by default	2019-12-24 13:27:42 +02:00
Aliaksandr Valialkin	29d2ce54cb	all: use gozstd instead of pure Go zstd for GOARCH=amd64	2019-12-24 12:43:59 +02:00
Aliaksandr Valialkin	afa8b34d27	Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner" This reverts commit `9f50232e70`. Reason to revert: this leaves incomplete logs on app shutdown.	2019-12-24 12:20:45 +02:00
Aliaksandr Valialkin	6358cf3d47	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:26 +02:00
Aliaksandr Valialkin	44f886cc9c	lib/encoding/zstd: typo fix	2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin	108a60d69e	lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines This should fix goroutine leak for https://github.com/klauspost/compress/issues/195	2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin	335bd0ac0a	lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.	2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin	ba17fcbcc5	deployment/docker: update docker image tags from v1.31.1-cluster to v1.31.2-cluster	2019-12-20 13:06:49 +02:00
Aliaksandr Valialkin	9f50232e70	lib/logger: prevent from blocking when log output isn't consumed in timely manner Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.	2019-12-20 11:49:42 +02:00
Aliaksandr Valialkin	cc8a1bae0e	app/vmselect: add `-search.maxExportDuration` command-line flag for limiting `/api/v1/export` duration Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275	2019-12-20 11:37:18 +02:00
Aliaksandr Valialkin	a37a006f11	lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores	2019-12-19 18:17:05 +02:00
Aliaksandr Valialkin	8d79412b26	lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series This should speed up `/api/v1/import` and make it more scalable on multi-core systems.	2019-12-19 15:13:36 +02:00
Aliaksandr Valialkin	8b56b849e9	app/vminsert: return StatusNoContent http response for `/api/v1/import` to be consistent with other insert handlers	2019-12-19 01:22:01 +02:00
Aliaksandr Valialkin	05ec8afb3a	lib/httpserver: sync the code with master branch	2019-12-18 23:08:32 +02:00
Aliaksandr Valialkin	a045c62532	docs/ExtendedPromQL.md: rewording regarding `scalar` vs `instant vector` difference	2019-12-18 21:47:31 +02:00
Aliaksandr Valialkin	cd04f6e82d	docs/Home.md: fix link to case studies	2019-12-18 01:05:15 +02:00
Aliaksandr Valialkin	4e8583bb02	docs: renaming: `PromQL extensions -> MetricsQL`	2019-12-18 00:57:53 +02:00
Aliaksandr Valialkin	198debc1c6	deployment/docker: update docker image tags from v1.30.6-cluster to v1.31.1-cluster	2019-12-16 01:49:00 +02:00
Aliaksandr Valialkin	6a185b7809	app/vmselect: add ability to pass `match[]`, `start` and `end` to `/api/v1/labels` This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`. See https://github.com/prometheus/prometheus/issues/6178 for more details.	2019-12-15 00:20:43 +02:00
Aliaksandr Valialkin	a7bf8e77af	app/vminsert: simultaneously accept `telnet put` and `HTTP /api/put` OpenTSDB metrics at `-opentsdbListenAddr` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:42:18 +02:00
Aliaksandr Valialkin	bc3984a5b3	lib/logger: add `-loggerFormat` for choosing log message formats Supported formats: default, json Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265	2019-12-13 15:09:18 +02:00
Aliaksandr Valialkin	aaf2545bdb	`make vendor-update`	2019-12-12 19:38:56 +02:00
Aliaksandr Valialkin	b238997a84	all: rename `Extended PromQL` to `PromQL extensions`	2019-12-12 19:29:59 +02:00
Aliaksandr Valialkin	bf8cf77694	docs: sync with master branch	2019-12-12 14:54:42 +02:00
Aliaksandr Valialkin	fef2eefb5e	docs: add Dreamteam numbers	2019-12-12 01:01:58 +02:00
Aliaksandr Valialkin	aad6ac76b9	docs/Single-server-VictoriaMetrics.md: sync with README.md	2019-12-12 00:56:04 +02:00
Aliaksandr Valialkin	cffaeda0f1	all: publish Docker images for the following GOARCH: amd64, arm, arm64, ppc64le and 386 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/258	2019-12-11 23:33:11 +02:00
Aliaksandr Valialkin	c25b97829f	app/vmselect/promql: return `lower` and `upper` bounds for the estimated percentile from `histogram_quantile` if third arg is passed Updates https://github.com/prometheus/prometheus/issues/5706	2019-12-11 14:00:18 +02:00
Aliaksandr Valialkin	557909aa81	docs: sync with master branch	2019-12-11 14:00:15 +02:00
Aliaksandr Valialkin	f79b61e2a1	app/vmselect/promql: return matrix instead of vector on subqueries to `/api/v1/query` like Prometheus does	2019-12-11 00:57:54 +02:00
Aliaksandr Valialkin	5d2ff573aa	app/vmselect/promql: allow negative offsets Updates https://github.com/prometheus/prometheus/issues/6282	2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin	c444a929a6	deployment/docker: update Docker image tags from v1.30.5-cluster to v1.30.6-cluster	2019-12-10 00:13:59 +02:00
Aliaksandr Valialkin	7edfa4d0cc	docs: use relative links	2019-12-09 23:05:39 +02:00
Aliaksandr Valialkin	e81a2bfdb3	docs: mention about `/api/v1/import` in Single-server-VictoriaMetrics.md	2019-12-09 23:05:38 +02:00
Aliaksandr Valialkin	033d252836	docs: mention about `/api/v1/import` in Cluster-VictoriaMetrics.md	2019-12-09 23:00:37 +02:00
Aliaksandr Valialkin	bd60dcb8ed	deployment/docker: update Go builder from go1.13.4 to go1.13.5	2019-12-09 22:57:43 +02:00
Aliaksandr Valialkin	c81a89a8ed	app/vminsert: add `/api/v1/import` handler Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6	2019-12-09 22:37:49 +02:00
Aliaksandr Valialkin	0c304439d4	app/vminsert: consistency renaming for counters	2019-12-09 16:44:26 +02:00
Aliaksandr Valialkin	3694efd005	lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.	2019-12-09 15:45:23 +02:00
Roman Khavronenko	924af22ced	#251 - add Logging rate panel (#255 )	2019-12-09 13:07:09 +02:00
Aliaksandr Valialkin	b809df03f8	vendor: fix broken build for GOARCH=arm64 on golang.org/x/sys/unix	2019-12-08 13:28:20 +02:00
Aliaksandr Valialkin	9442e619ea	vendor: fix arm build for golang.org/x/sys/unix/zptrace_armnn_linux.go	2019-12-08 12:49:21 +02:00
Aliaksandr Valialkin	c217a53c35	`make vendor-update`	2019-12-07 23:11:26 +02:00
Aliaksandr Valialkin	3534e71c96	app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689	2019-12-07 23:00:54 +02:00
Aliaksandr Valialkin	8cf015c34f	README.md: mention that VictoriaMetrics is built on `shared nothing` architecture	2019-12-05 20:38:20 +02:00
Aliaksandr Valialkin	7a775714ab	deployment/docker: update Docker image tags from v1.30.4-cluster to v1.30.5-cluster	2019-12-05 20:15:50 +02:00
Aliaksandr Valialkin	e243429b39	docs: add docs as in the single-node branch	2019-12-05 19:28:29 +02:00
Aliaksandr Valialkin	d39bba3547	app/vmselect/promql: add `{topk\|bottomk}_{min\|max\|avg\|median}` aggregate functions for returning the exact k time series on the given time range The full list of functions added: - `topk_min(k, q)` - returns top K time series with the max minimums on the given time range - `topk_max(k, q)` - returns top K time series with the max maximums on the given time range - `topk_avg(k, q)` - returns top K time series with the max averages on the given time range - `topk_median(k, q)` - returns top K time series with the max medians on the given time range - `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range - `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range - `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range - `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range	2019-12-05 19:27:45 +02:00
Aliaksandr Valialkin	639967db59	lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after `runTransactions` call. `runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction deletions are finished before returning from `runTransactions`.	2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin	7c0dd85a7c	lib/httpserver: add `/ping` handler for compatibility with Influx agents Certain Influx agents check for `/ping` endpoint before starting to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint	2019-12-04 19:18:18 +02:00
Aliaksandr Valialkin	877b83ce97	deployment/docker: update docker image tags from v1.30.3-cluster to v1.30.4-cluster	2019-12-04 01:53:04 +02:00
Aliaksandr Valialkin	e0f43e1f66	app/vmselect: add placeholders for `/api/v1/rules` and `/api/v1/alerts`	2019-12-03 19:38:09 +02:00
Aliaksandr Valialkin	534da0a8c3	lib/storage: fall back to global inverted index if a filter match too many time series in per-day index Previously this resulted to error message. The query may succeed via search in global index.	2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin	6eb698d1cc	lib/storage: fix printing tag filters in TagFilters.String	2019-12-03 14:25:20 +02:00
Aliaksandr Valialkin	c04f60db35	lib/storage: print `__name__` instead of empty string in user-visible tag filters	2019-12-03 14:18:18 +02:00
Aliaksandr Valialkin	625f6ca761	lib/storage: optimize regexp filter search	2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin	47077c02ba	deployment/docker: update image tags from v1.30.2-cluster to v1.30.3-cluster	2019-12-02 22:51:31 +02:00
Aliaksandr Valialkin	6bee9115aa	vendor: update github.com/VictoriaMetrics/metrics from v1.9.1 to v1.9.2 This fixes possible deadlock when metrics.WritePrometheus calls Gauge callback, which calls metrics functions with internal lock.	2019-12-02 22:31:47 +02:00
Aliaksandr Valialkin	b9616c017f	lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed This should fix the issue on NFS when incompletely removed dirs may be left after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction files are already removed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin	4e22b521c2	lib/storage: remove metricID with missing metricID->metricName entry The metricID->metricName entry can be missing in the indexdb after unclean shutdown when only a part of entries for new time series is written into indexdb. Recover from such a situation by removing the broken metricID. New metricID will be automatically created for time series with the given metricName when new data point will arive to it.	2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin	387f62f468	deployment/docker: update docker image tag from v1.30.1-cluster to v1.30.2-cluster	2019-12-02 15:17:41 +02:00
Aliaksandr Valialkin	5a62415bec	lib/storage: protect from time drift during indexdb rotation Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248	2019-12-02 14:43:11 +02:00
Aliaksandr Valialkin	cf85c567d1	lib/logger: merge `file` and `line` labels into `location="file:line"` This should improve the usability for `vm_log_messages_total` metric during practical queries	2019-12-02 14:43:09 +02:00
Aliaksandr Valialkin	f055dbefda	lib/storage: generate more human-friendly result in TagFilters.String	2019-12-02 13:56:40 +02:00
Aliaksandr Valialkin	819bb36852	app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals This should improve scrape interval estimation for tiem series with gaps.	2019-12-02 13:43:04 +02:00
Aliaksandr Valialkin	29f39f866e	lib/logger: consistency renaming from `vm_log_messages_count` to `vm_log_messages_total`, since this is a counter	2019-12-02 00:47:12 +02:00
Aliaksandr Valialkin	15eaff1745	lib/logger: track the number of log messages by `(level, file, line)` in the `vm_log_messages_count` metric	2019-12-01 18:38:30 +02:00
Aliaksandr Valialkin	d456ec7589	lib/netutil: use IPv6 for both listening and dialing if `-enabledTCP6` is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244	2019-12-01 02:52:53 +02:00
Aliaksandr Valialkin	1595dcd3d9	app/vminsert/influx: allow empty measurement in Influx line protocol In this case metric names are mapped directly from field names without any prefixes.	2019-11-30 21:59:07 +02:00
Aliaksandr Valialkin	1e2019b1b6	app/vmselect/promql: fix corner case for `increase` over time series with gaps In this case `increase` could return invalid high value for the first point after the gap.	2019-11-30 01:34:18 +02:00
Aliaksandr Valialkin	4c63caa37c	deployment/docker/certs: update TLS certs source from alpine:3.9 to alpine:3.10	2019-11-29 19:55:36 +02:00
Aliaksandr Valialkin	274d8bcb7b	deployment/docker/docker-compose.yml: remove superflouos volume mount (#246 ) The `provisioning/dashboards` folder should be already mounted on the previous line. This should fix the `/bin/sh: can't create dashboards/vm.json: Permission denied` error on `docker-compose up`	2019-11-29 18:10:50 +02:00
Aliaksandr Valialkin	7e734433a3	lib/backup: cosmetic fixes after #243	2019-11-29 18:07:41 +02:00
glebsam	4a192cb832	Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243 ) * Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile * make fmt	2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin	4810f1dde6	lib/netutil: add `-enableTCP6` command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports	2019-11-29 17:33:07 +02:00
Aliaksandr Valialkin	93dbec971b	deployment/docker: update docker image tags from v1.30.0-cluster to v1.30.1-cluster	2019-11-28 22:26:25 +02:00
Aliaksandr Valialkin	90f2530f9f	README.md: add `monitoring` section	2019-11-28 19:16:05 +02:00
Aliaksandr Valialkin	409c939621	lib/backup: remove `flock.lock` file in empty dirs This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations: 1. Stop VictoriaMetrics. 2. Delete `<-storageDataPath>` dir. 3. Start VictoriaMetrics, then stop it. 4. Restore data from backup with `vmrestore`. 5. Start VictoriaMetrics. `vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.	2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin	572fe61857	README.md: remove the unnecessary step during restoring from backups	2019-11-27 19:56:15 +02:00
Aliaksandr Valialkin	396ed27759	vendor: `make vendor-update`	2019-11-27 15:34:18 +02:00
Aliaksandr Valialkin	2571903522	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.2 to v1.5.4	2019-11-27 15:31:36 +02:00
Aliaksandr Valialkin	093f94d2db	deployment/docker: update Grafana from v6.4.4 to v6.5.0	2019-11-27 15:10:01 +02:00
Aliaksandr Valialkin	8ccbcaf99f	deployment/docker: update image tags from v1.29.5-cluster to v1.30.0-cluster	2019-11-27 14:54:21 +02:00
Aliaksandr Valialkin	def9ccd360	app/vmselect/prometheus: consistently apply `nocache` arg to `/api/v1/query` the same way ast to `/api/v1/query_range` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241	2019-11-26 22:55:50 +02:00
Aliaksandr Valialkin	e0ac068112	app/vmselect/prometheus: fix content-type for `/api/v1/export` responses The correct Content-Type should be `application/stream+json` instead of `application/json` Thanks to Joshua Ryder for pointing to this.	2019-11-26 17:44:27 +02:00
Aliaksandr Valialkin	28cc4c09b5	app/vmselect/promql: remove zero timeseries from `prometheus_buckets` output	2019-11-25 19:10:13 +02:00
Aliaksandr Valialkin	8811bec14e	app/vmselect/prometheus: reduce default value for `-search.latencyOffset` from 60s to 30s 30 seconds should be enough for almost all the cases	2019-11-25 16:33:36 +02:00
Aliaksandr Valialkin	f7da9b2db2	app/vmselect/promql: allow nested parens	2019-11-25 16:13:33 +02:00
Aliaksandr Valialkin	d2619d6dce	vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1	2019-11-25 15:22:50 +02:00
Aliaksandr Valialkin	f46fb6c740	app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph This should reduce the amounts memory allocations	2019-11-25 14:24:30 +02:00
Aliaksandr Valialkin	0f184affa7	app/vmselect/promql: optimize binary search over big number of samples during rollup calculations	2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin	dbd07041ae	app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0	2019-11-25 13:44:08 +02:00
Aliaksandr Valialkin	406e36f817	vendor: update github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0	2019-11-25 13:19:34 +02:00
Aliaksandr Valialkin	8bb254d960	app/vmselect/promql: add `histogram` aggregate function, which is useful for building heatmaps from multiple time series	2019-11-24 00:04:15 +02:00
Aliaksandr Valialkin	e70f543321	vendor: update github.com/VictoriaMetrics/metrics from v1.8.2 to v1.8.3	2019-11-24 00:04:14 +02:00
Aliaksandr Valialkin	d24fc87a6f	lib/decimal: calculate ln2/ln10 constant during compile time	2019-11-23 15:52:39 +02:00
Aliaksandr Valialkin	414259f47b	app/vmselect/promql: do not take into account buckets with negative counters in `prometheus_buckets`	2019-11-23 14:19:19 +02:00
Aliaksandr Valialkin	193d553f6d	app/vmselect/promql: properly handle `histogram_quantile(0, ...)` with zero buckets	2019-11-23 14:02:25 +02:00
Aliaksandr Valialkin	f8298c7f13	app/vmselect: add `vm_per_query_{rows,series}_processed_count` histograms	2019-11-23 13:23:03 +02:00
Aliaksandr Valialkin	b1c3284fd0	dashboards: remove deprecated dashboards - now only victoriametrics.json is officially supported	2019-11-23 12:43:38 +02:00
Aliaksandr Valialkin	654473f6c6	vendor: update github.com/VictoriaMetrics/metrics from v1.8.1 to v1.8.2	2019-11-23 11:49:18 +02:00
Aliaksandr Valialkin	4d76977745	app/vmselect/promql: transparently apply `prometheus_buckets` in `histogram_quantile`	2019-11-23 11:49:16 +02:00
Aliaksandr Valialkin	cfeb606e73	vendor: update github.com/VictoriaMetrics/metrics from v1.8.0 to v1.8.1	2019-11-23 00:48:55 +02:00
Aliaksandr Valialkin	2af7ca1122	vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms	2019-11-23 00:21:57 +02:00
Aliaksandr Valialkin	5f6f03c692	app/vmselect/promql: add `prometheus_buckets` function for converting the upcoming histogram buckets from `github.com/VictoriaMetrics/metrics` to Prometheus-compatible buckets	2019-11-23 00:21:56 +02:00
Aliaksandr Valialkin	17d08c1fe0	app/vmselect: adjust `end` arg instead of adjusting `start` arg if start > end `start` arg has higher chances to be set properly comparing to `end` arg, so it is expected that the `end` arg could be adjusted if it was set incorrectly.	2019-11-22 16:12:53 +02:00
Aliaksandr Valialkin	14ba958e9a	deployment/docker: update docker image tag from v1.29.3-cluster to v1.29.5-cluster	2019-11-22 14:05:17 +02:00
Aliaksandr Valialkin	7c48f8611f	vendor: updated github.com/valyala/gozstd from v1.6.2 to v1.6.3	2019-11-21 23:56:27 +02:00
Aliaksandr Valialkin	b9e53490b9	lib/storage: move non-matching tag filters to the top at matchTagFilters This should reduce the amount of useless work needed for matching the next metricNames.	2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin	33d9d63393	lib/storage: speed up time series search for queries with multiple filters Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.	2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin	926290d73e	Makefile: create files with sha256 checksums during `make release` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/19	2019-11-20 22:45:23 +02:00
Aliaksandr Valialkin	a02a57fbe9	lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers	2019-11-20 15:40:03 +02:00
Aliaksandr Valialkin	3d1f4408cf	lib/decimal: increase decimal->float speed conversion for integer numbers	2019-11-20 14:09:10 +02:00
Aliaksandr Valialkin	f1f2eff08f	lib/decimal: reduce rounding error when converting from decimal to float with negative exponent While at it, slightly increase the conversion performance by moving fast path to the top of the loop.	2019-11-19 23:34:41 +02:00
Aliaksandr Valialkin	2929a41e3b	`make vendor-update`	2019-11-19 21:34:44 +02:00
Aliaksandr Valialkin	17eca31989	lib/backup: retrieve only the required metadata when reading GCS objects	2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin	ccf3d143c5	`make vendor-update`	2019-11-19 21:30:49 +02:00
Aliaksandr Valialkin	216a260ced	app/{vmbackup,vmrestore}: add `-maxBytesPerSecond` command-line flag for limiting the used network bandwidth during backup / restore	2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin	9d1ee1e2ae	lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore	2019-11-19 18:35:59 +02:00
Aliaksandr Valialkin	5ae47e8940	app/vmselect/prometheus: properly adjust too big time `time` on `/api/v1/query` Too big `time` must be adjusted to `now()-queryOffset`.	2019-11-19 00:42:07 +02:00
Aliaksandr Valialkin	6ca4b94511	lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues The previous commit was accidentally creating 10x smaller number of time series than Prometheus and this led to invalid benchmark results. The updated benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 6194893 -97.73% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 10781372 -92.19% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 10632834 -92.11% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 10679975 -94.55% BenchmarkHeadPostingForMatchers/i=~"." 7962582919 100118510 -98.74% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 154955671 -97.96% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 258003769 -77.42% BenchmarkHeadPostingForMatchers/i!="" 9964150263 159783895 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo" 216995884 10937895 -94.96% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo" 202541348 10990027 -94.57% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 87004349 -82.11% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 53342793 -84.79% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 54256156 -85.76% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 21823279 -75.62% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 46671359 -87.70% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo" 424563825 53915842 -87.30% VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)	2019-11-18 19:48:27 +02:00
Aliaksandr Valialkin	6f61fd367a	lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus See the corresponding benchmark in Prometheus - `23c0299d85/tsdb/head_bench_test.go (L52)` The benchmark allows performing apples-to-apples comparison of time series search in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness - contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix it. Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots: - Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers - VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers Benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 364977 -99.87% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 1181636 -99.14% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 1141578 -99.15% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 1148056 -99.41% BenchmarkHeadPostingForMatchers/i=~"." 7962582919 8716755 -99.89% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 12096587 -99.84% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 16164560 -98.59% BenchmarkHeadPostingForMatchers/i!="" 9964150263 12230021 -99.88% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo" 216995884 1173476 -99.46% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo" 202541348 1299743 -99.36% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 11555193 -97.62% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 5607506 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 6380335 -98.32% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 2078970 -97.68% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 6561368 -98.27% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo" 424563825 6757132 -98.41% The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics. Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.	2019-11-18 18:47:02 +02:00
Aliaksandr Valialkin	77bb66a5be	app/vmselect/promql: properly calculate `integrate(q[d])`	2019-11-13 21:11:03 +02:00
Aliaksandr Valialkin	c33640664a	app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235	2019-11-13 20:26:07 +02:00
Aliaksandr Valialkin	d297b65089	lib/storage: add `vm_cache_size_bytes{type="storage/hour_metric_ids"}` metric	2019-11-13 20:26:05 +02:00
Aliaksandr Valialkin	31376fd353	deployment/docker: update docker image tag from v1.29.2-cluster to v1.29.3-cluster	2019-11-13 18:32:08 +02:00
Aliaksandr Valialkin	494ad0fdb3	lib/storage: remove inmemory index for recent hour, since it uses too much memory Production workload shows that the index requires ~4Kb of RAM per active time series. This is too much for high number of active time series, so let's delete this index. Now the queries should fall back to the index for the current day instead of the index for the recent hour. The query performance for the current day index should be good enough given the 100M rows/sec scan speed per CPU core.	2019-11-13 18:08:58 +02:00
Aliaksandr Valialkin	90bde025f0	deployment/docker: update image tag from v1.29.0-cluster to v1.29.2-cluster	2019-11-13 15:24:44 +02:00
Aliaksandr Valialkin	633dd81bb5	lib/storage: add `-disableRecentHourIndex` flag for disabling inmemory index for recent hour This may be useful for saving RAM on high number of time series aka high cardinality	2019-11-13 15:10:12 +02:00
Aliaksandr Valialkin	f1620ba7c0	lib/storage: fix inmemory inverted index issues found in v1.29 Issues fixed: - Slow startup times. Now the index is loaded from cache during start. - High memory usage related to superflouos index copies every 10 seconds.	2019-11-13 13:35:38 +02:00
Aliaksandr Valialkin	87b39222be	Revert "lib/fs: do not postpone directory removal on NFS error" This reverts commit 21aeb02b46649ac9906cb37733f7b155a77a0db9.	2019-11-12 16:29:50 +02:00
Mike Poindexter	955a592106	Add test for invalid caching of tsids (#232 ) * Add test for invalid caching of tsids * Clean up error handling	2019-11-12 15:52:46 +02:00
Roman Khavronenko	ce8cc76a42	add links and fix cache metric name (#233 )	2019-11-12 15:06:56 +02:00
Aliaksandr Valialkin	6afb7a50a9	deployment/docker: upgrade Grafana release from v6.4.3 to v6.4.4	2019-11-12 03:50:54 +02:00
Aliaksandr Valialkin	5b677a57e3	deployment/docker: upgrade Go from v1.13.4 to v1.13.4	2019-11-12 03:49:07 +02:00
Aliaksandr Valialkin	d420871d79	deployment/docker: upgrade docker image tag from v1.28.3-cluster to v1.29.0-cluster	2019-11-12 03:44:45 +02:00
Aliaksandr Valialkin	584d8362c8	deployment: update Prometheus from v2.13.0 to v2.14.0	2019-11-12 03:43:59 +02:00
Roman Khavronenko	828f0a2a4b	prepare dashboard for external sharing (#231 )	2019-11-12 00:23:24 +02:00
Oleg Kovalov	74ba42d111	fix misspelled words (#229 )	2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin	c48e39eea9	lib/storage: add tests for dateMetricIDCache	2019-11-11 13:21:05 +02:00
Aliaksandr Valialkin	bdc9045485	README.md: mention that replication doesnt save from disaster	2019-11-11 00:58:08 +02:00
Aliaksandr Valialkin	01801e9e03	dashboards: there will no 1.28.4 release. It will be 1.29.0	2019-11-10 22:05:10 +02:00
Aliaksandr Valialkin	6bdde0d6d4	lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has	2019-11-10 22:04:23 +02:00
Roman Khavronenko	7247a7862d	add description, churn rate panel, storage.ingestion rate panel (#228 )	2019-11-10 20:32:10 +02:00
Aliaksandr Valialkin	5f52eb7653	lib/fs: do not postpone directory removal on NFS error Continue trying to remove NFS directory on temporary errors for up to a minute. The previous async removal process breaks in the following case during VictoriaMetrics start - VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them. - The transaction instructs removing old directories for parts, which were already merged into bigger part. - VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors. - VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished and finds directories, which should be removed, but weren't still removed due to NFS errors. - VictoriaMetrics panics when it finds unexpected empty directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-11-10 13:27:16 +02:00
Aliaksandr Valialkin	9ea2bd822e	lib/storage: implement per-day inverted index	2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin	5d8de72414	app/vmrestore: the upcoming release would be 1.29.0	2019-11-10 00:20:18 +02:00
Aliaksandr Valialkin	dea2f3efed	lib/storage: use specialized cache for (date, metricID) entries This improves ingestion performance.	2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin	9a43902bd8	lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero	2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin	c16e17dede	lib/storage: properly set time range when deleting time series	2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin	8126007c15	lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster	2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin	50773348d3	lib/storage: small code prettifying	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	44fa8226df	lib/uint64set: remove superflouos check for item existence before deleting it in Set.Subtract	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	0bc54c23ce	lib/storage: inmemoryInvertedIndex prettifying	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	46e67bb78c	lib/storage: export `vm_new_timeseries_created_total` metric for determining time series churn rate	2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin	0063c857f5	lib/storage: add inmemory inverted index for the last hour It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.	2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin	33abbec6b4	app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions Incremental aggregate functions don't keep all the selected time series in memory - they keep only up to GOMAXPROCS time series for incremental aggregations. Take into account that the number of time series in RAM can be higher if they are split into many groups with `by (...)` or `without (...)` modifiers. This should reduce the number of `not enough memory for processing ... data points` false positive errors.	2019-11-08 19:37:43 +02:00
Aliaksandr Valialkin	7d7fbf890e	app/{vmbackup,vmrestore}: add `vmbackup` and `vmrestore` tools for creating backups on s3 or gcs from instant snapshots Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38	2019-11-07 21:26:43 +02:00
Roman Khavronenko	4e7a2a41a4	Cluster dashboard (#222 ) * add dashboard for cluster version * fix queries and panels * review fixes * use resident memory for memory usage panel * fix job selectors	2019-11-07 12:09:27 +02:00
Aliaksandr Valialkin	89c03a5464	lib/storage: populate partition names from both `small` and `big` directories Certain partition directories may be missing after restoring from backups if they had no data. Re-create such directories on start.	2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin	1c777e0245	lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter The origin of the error has been detected and documented in the code, so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`, so it could be monitored and alerted on high error rates. Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`, so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.	2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin	c567a4353a	lib/storage: take into account the requested time range when caching TSIDs for the given tag filters	2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin	c6564c5d26	lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting	2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin	2ef5082ead	deployment/docker: update docker images from v1.28.2-cluster to v1.28.3-cluster	2019-11-05 18:08:50 +02:00
Aliaksandr Valialkin	a10c4cad85	lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions	2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin	e5b1fa0c38	lib/storage: separate the max inverted index scan loops per metric into fast and slow loops Slow loops could require seeks and expensive regexp matching, while fast loops just scans all the metricIDs for the given `tag=value` prefix. So these operations must have separate max loops multiplier.	2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin	f93c4f2493	lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive This should improve performance for query filters over big number of time series.	2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin	f48e97263c	lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name The new value reduces the amount of wasted work during index scans over big number of time series.	2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin	d2f688c550	lib/storage: try potentially faster tag filters at first, then apply slower tag filters The fastest tag filters are non-negative non-regexp, since they are the most specific. The slowest tag filters are negative regexp, since they require scanning all the entries for the given label.	2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin	a72b22a8b1	Makefile: add `pprof-cpu` rule for inspecting CPU profiles with `PPROF_FILE=/path/to/cpu.pprof make pprof-cpu`	2019-11-04 12:43:57 +02:00
Aliaksandr Valialkin	2a38d30f93	lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method This fixes lint errors	2019-11-04 01:06:45 +02:00
Artem Navoiev	e05500cbd4	add unittests for bytesutil and storage (#221 )	2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin	f5fbc3ffd7	lib/{storage,uint64set}: add Set.Union() function and use it	2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin	23e078261e	lib/storage: tune the returned value from adjustMaxMetricsAdaptive	2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin	386c349c8c	lib/storage: remove interface conversion in hot path during block merging This should improve merge speed a bit for parts with big number of small blocks.	2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin	26ffc77622	lib/{storage,mergeset}: create missing partition directories after restoring from backups Backup tools could skip empty directories. So re-create such directories on the first run.	2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin	5d439cc6f2	lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.	2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin	1037053fed	lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones Time series with such values are quite common in monitoring space, so it would be great to have benchmarks for them.	2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin	46b8e13d8c	deployment/docker: update image tag from v1.28.1-cluster to v1.28.2-cluster	2019-11-01 16:33:46 +02:00
Aliaksandr Valialkin	44fab198e2	lib/uint64set: return an emptry set instead of nil set from `Set.Clone`, since the caller may add data to the cloned set This fixes the following panic in v1.28.1: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e] goroutine 1155 [running]: github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(Set).Add(0x0, 0x15b3bfb41e8b71ec) github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000) github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260 github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)	2019-11-01 16:12:21 +02:00
Aliaksandr Valialkin	4a8251feff	app/vmselect/promql: add `lag(q[d])` function, which returns the lag between the current timestamp and the timstamp for the last data point in `q`	2019-11-01 12:21:43 +02:00
Aliaksandr Valialkin	bd065aad5e	deployment/docker: update docker images from v1.28.0-cluster to v1.28.1-cluster	2019-10-31 17:03:27 +02:00
Aliaksandr Valialkin	6ab9c98a1e	app/vmstorage: add `-bigMergeConcurrency` and `-smallMergeConcurrency` flags for tuning the maximum number of CPU cores used during merges	2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin	6a22727676	lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups	2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin	ca480915ca	lib/storage: small cleanup in Storage.add	2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin	22030b558f	lib/decimal: speed up FromFloat for common case with integers	2019-10-31 13:25:09 +02:00
Aliaksandr Valialkin	6510258a80	lib/decimal: increase float64->decimal conversion precision a bit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-30 02:04:28 +02:00
Aliaksandr Valialkin	a27e034a40	lib/storage: get parts to merge after applying the limit on the number of concurrent merges This should reduce write amplification under high ingestion rate.	2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin	5d2276dbf7	lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge	2019-10-29 12:45:37 +02:00
Aliaksandr Valialkin	78166cc478	vendor: update github.com/VictoriaMetrics/fastcache from v1.5.1 to v1.5.2	2019-10-29 11:31:36 +02:00
Aliaksandr Valialkin	f581b2736a	lib/fs: typo fix in comment to WriteFileAtomically	2019-10-29 11:31:34 +02:00
Aliaksandr Valialkin	a638c6d4f8	vendor: `make vendor-update`	2019-10-28 13:41:13 +02:00
Roman Khavronenko	1750ee1575	* #215 : update klauspost/compress lib * #215: bump klauspost/compress lib to 1.9.1	2019-10-28 13:41:10 +02:00
Aliaksandr Valialkin	eb513e7ba3	lib/decimal: increase float->decimal conversion precision for big numbers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-28 13:23:54 +02:00
Aliaksandr Valialkin	4e6bf6f538	app/vmselect: add `-search.latencyOffset` flag for tuning the time after data collection when data points become visible in query results Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218	2019-10-28 12:32:36 +02:00
Aliaksandr Valialkin	121be98325	deployment/docker: upgrade Go builder from go1.13.1 to go1.13.3	2019-10-20 23:49:18 +03:00
hanzai	52778da1f3	warns during rows addition (#214 )	2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin	6823aaaf08	README.md: add `capacity planning` chapter	2019-10-19 10:48:00 +03:00
Aliaksandr Valialkin	78fc35c9b1	all: `make fmt`	2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin	88d793305d	Makefile: disable `structcheck` in `golangci-lint`, since it gives false positive on embedded structs	2019-10-17 20:00:17 +03:00
Aliaksandr Valialkin	5b01b7fb01	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	5d2af2cfa2	vendor: update github.com/valyala/quicktemplate from v1.2.0 to v1.3.1	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	12c8afc3f2	lib/memory: properly handle int overflow in sysTotalMemory This should fix builds on 32-bit architectures such as arm. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	7d7d7a7d4e	deployment/docker/docker-compose.yml: update Prometheus from v2.12.0 to v2.13.0	2019-10-16 12:39:17 +03:00
Aliaksandr Valialkin	e0109fc316	deployment/docker/docker-compose.yml: update VictoriaMetrics image from v1.27.0-cluster to v1.28.0-cluster	2019-10-16 12:39:17 +03:00
Aliaksandr Valialkin	469d169a5d	README.md: mention our Slack	2019-10-16 12:31:53 +03:00
Aliaksandr Valialkin	99786c2864	app/vmselect/prometheus: add `-search.maxLookback` command-line flag for overriding dynamic calculations for max lookback interval This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically from interval between datapoints for each input time series if the flag isn't set. The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209	2019-10-15 21:37:17 +03:00
Aliaksandr Valialkin	ce266d157d	README.md: mention contact email for consulting and support	2019-10-15 00:11:39 +03:00
Aliaksandr Valialkin	dc2f822577	lib/prompb: removed outdated README.md	2019-10-14 22:16:36 +03:00
Aliaksandr Valialkin	8ecdb04b7c	Makefile: remove obsolete Makefile include from /helm/ directory	2019-10-13 23:22:54 +03:00
Aliaksandr Valialkin	92e0ca6bbf	vendor: `make vendor-update`	2019-10-13 23:18:28 +03:00
Artem Navoiev	75504747c8	Update README.md change helm section	2019-10-13 23:05:37 +03:00
Artem Navoiev	3d3d87f718	[deployment] remove helm chart	2019-10-13 23:03:52 +03:00
Aliaksandr Valialkin	bf6fe234b2	README.md: mention about `delete_series` handler Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/207	2019-10-10 02:09:38 +03:00
Aliaksandr Valialkin	f1a7965676	README.md: refer to comment about ingestion rate scalability Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-10-09 17:28:25 +03:00
Aliaksandr Valialkin	7b6570489a	README.md: add a few words about scalability	2019-10-09 13:01:59 +03:00
Aliaksandr Valialkin	661b8ede5b	lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order	2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin	7f4a04ee6a	lib/storage: typo fix in comment to maxRowsPerSmallPart.	2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin	7e410e1412	lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.	2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin	a5302a6651	app/vmselect/promql: take into account the previous point when calculating `max_over_time` and `min_over_time` This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns the minimum value. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204	2019-10-08 12:30:16 +03:00
Aliaksandr Valialkin	95d0f1bfd1	vendor: `make vendor-update`	2019-10-06 15:48:45 +03:00
Aliaksandr Valialkin	84b3b29644	README.md: add `binaries` section with urls to pre-compiled binaries and docker images	2019-10-06 11:42:36 +03:00
Aliaksandr Valialkin	39b18b1dcd	vendor: update github.com/VictoriaMetrics/metrics from v1.7.1 to v1.7.2	2019-10-06 11:20:03 +03:00
Stian Ovrevage	ef6e01b1fa	Add bool to extraLabels. Fix tls indentation	2019-10-03 21:47:00 +03:00
Stian Ovrevage	4fb63d7d61	Fix helm template indentation	2019-10-03 21:47:00 +03:00
Aliaksandr Valialkin	9fce611fbb	lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195	2019-09-30 12:27:30 +03:00
Aliaksandr Valialkin	483af3a97a	app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially This became the case after `b7ee2e7af2` .	2019-09-30 00:13:33 +03:00
Aliaksandr Valialkin	946ca438a6	app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock This also allows marshaling outside lock, thus reducing the amount of work under the lock.	2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin	e92e39eddf	app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series	2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin	56dff57f77	app/vmselect/netstorage: reduce memory usage when fetching big number of data blocks from vmstorage Dump data blocks directly to temporary file instead of buffering them in RAM	2019-09-28 12:21:57 +03:00
Aliaksandr Valialkin	ba460f62e6	app/vmselect/promql: do not generate timestamps for NaN values in `timestamp` function according to Prometheus logic	2019-09-27 18:55:16 +03:00
Stian Øvrevåge	a9dac3829e	Update README.md - Fix helm command typos `victoria-mertrics` -> `victoria-metrics` in helm commands.	2019-09-27 12:33:48 +03:00
Aliaksandr Valialkin	de919574a5	deployment/docker: switch Go builder image from v1.13.0 to v1.13.1	2019-09-26 17:12:52 +03:00
Aliaksandr Valialkin	d0b4590099	lib/storage: optimize TSID comparison	2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin	95e3d648cb	lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.	2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin	2b8358726f	lib/storage: properly match labels against regexp with `(?i)` flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin	bd1cf053f6	app/vmselect/promql: add `increases_over_time` and `decreases_over_time` functions `increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`. `decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.	2019-09-25 20:38:51 +03:00
Aliaksandr Valialkin	4e3871ac1e	lib/storage: add missing break in removeDuplicateMetricIDs	2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin	4468f9f966	lib/storage: remove duplicate MetricIDs in `tag->metricIDs` items before writing them into inverted index	2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin	adc18c3ee6	lib/{mergeset,storage}: do not cache inverted index blocks containing `tag->metricIDs` items This should reduce the amounts of used RAM during queries with filters over big number of time series.	2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin	8d398af92f	app/vminsert/netstorage: mention the data size that cannot be sent to vmstorage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-09-25 12:53:41 +03:00
Aliaksandr Valialkin	73ac7b8dd6	app/vminsert/netstorage: make sure the conn exists before closing it in storageNode.closeBrokenConn The conn can be missing or already closed during the call to storageNode.closeBrokenConn. Prevent `nil pointer dereference` panic by verifying whether the conn is already closed. Thanks to @CH-anhngo for reporting the issue. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189	2019-09-25 10:36:50 +03:00
Aliaksandr Valialkin	c64fb91a43	lib/uint64set: optimize Set.AppendTo	2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin	de0e4eee2c	lib/storage: create and use `lib/uint64set` instead of `map[uint64]struct{}` This should improve inverted index search performance for filters matching big number of time series, since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls. See the corresponding benchmarks in `lib/uint64set`.	2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin	2212d0e421	lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows	2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin	9307de1b92	lib/storage: limit the number of metricIDs in tag->metricIDs row This reduces the overhead on index and metaindex in lib/mergeset	2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin	7734fc8012	lib/storage: share tsids across all the partSearch instances This should reduce memory usage when big number of time series matches the given query.	2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin	67a2bcb98a	lib/{storage,mergeset}: verify PrepareBlock callback results Do not touch the first and the last item passed to PrepareBlock in order to preserve sort order of mergeset blocks.	2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin	3304dc1e85	lib/mergeset: detect whether we are in test by executable suffix	2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin	d2ed8cb0b2	lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID The first item from each mergeset block goes into index (lib/mergeset.blockHeader), so it must be short in order to reduce index size.	2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin	0a9cb6368e	lib/workingsetcache: remove data race when resetting c.misses	2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin	7d13c31566	lib/{storage,mergeset}: merge `tag->metricID` rows into `tag->metricIDs` rows for common `tag` values This should improve lookup performance if the same `label=value` pair exists in big number of time series. This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows occupy less space than the original `tag->metricID` rows.	2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin	272e2f77c9	lib/encoding: optimize UnmarshalUint* and UnmarshalInt*	2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin	7e0c6d4ca6	lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries and MetricID->TSID entries are usually shorter than tag->MetricID entries. This should improve performance when selecting all the metricIDs.	2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin	b0c738ae8b	app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls	2019-09-19 22:16:11 +03:00
Aliaksandr Valialkin	bf8505353a	lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts	2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin	ebbef20535	lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations	2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin	89234f395d	lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs	2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin	6e586fa09c	lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects	2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin	410f993bf6	lib/mergeset: fill partHeader.firstItem on first block flush	2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin	c05885fb5f	lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless This will save CPU on subsequent calls for this filter	2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin	e041a196a7	deployment/docker/docker-compose.yml: update Prometheus from v2.3.2 to v2.12.0	2019-09-18 18:30:02 +03:00
Aliaksandr Valialkin	db71c940ea	lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there	2019-09-17 23:18:37 +03:00
Artem Navoiev	ccb6dc6925	[deployment] clean up helm (#185 )	2019-09-16 21:58:16 +03:00
Aliaksandr Valialkin	491b1317f4	vendor: update github.com/valyala/gozstd from v1.6.1 to v1.6.2	2019-09-16 21:50:02 +03:00
Aliaksandr Valialkin	5666112de2	deployment: switch docker image tag from v1.27.2-cluster to v1.27.3-cluster	2019-09-14 11:33:18 +03:00
Aliaksandr Valialkin	ba21622b78	vendor: `make vendor-update`	2019-09-13 22:49:34 +03:00
Aliaksandr Valialkin	020341d13a	deployment/docker: remove file system paths from the compiled binary	2019-09-13 22:46:07 +03:00
Aliaksandr Valialkin	550a12415a	app/vminsert/netstorage: log network errors when sending data to vmstorage nodes Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-09-13 22:26:24 +03:00
Aliaksandr Valialkin	41ef6b060e	lib/mergeset: properly check for sorted block headers Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181	2019-09-13 21:59:38 +03:00
Aliaksandr Valialkin	ee4585db33	app/vmselect/promql: properly handle subqueries like `aggr_func(rollup_func(metric[window:step]))` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184	2019-09-13 21:42:11 +03:00
hanzai	08cde5e3f4	lib/workingsetcache: adjust switching from mode=`split` to mode=`whole` smoothly and load cachefile successfully	2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin	828e5f6d26	app/vmselect/promql: binary operation fixes according to Prometheus behaviour The follosing issues were fixed: - VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers - VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list	2019-09-13 17:43:09 +03:00
Artem Navoiev	62b424bc4c	[ci] github actions - run pipeline on pull request. Fix running of test in external PR from forks	2019-09-11 14:54:45 +03:00
Aliaksandr Valialkin	ed50b8792b	app/vminsert/netstorage: reduce the maximum buffer size for rerouted rows, so it occupies less RAM	2019-09-11 14:50:30 +03:00
Aliaksandr Valialkin	b101064f8b	all: report the number of bytes read on io.ReadFull error This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin	2f4c950fe9	app/vminsert/netstorage: send per-storageNode bufs to vmstorage nodes in parallel This should improve the maximum ingestion throughput. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-09-11 14:50:19 +03:00
Aliaksandr Valialkin	694cc59ed1	app/vminsert/netstorage: dynamically adjust timeouts for sending packets from vminsert to vmstorage depending on packet size Bigger packets will have more chances to be sent to vmstorage.	2019-09-11 14:50:14 +03:00
Aliaksandr Valialkin	568ff61dcf	lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache	2019-09-09 11:42:45 +03:00
Artem Navoiev	dc6e4151b0	[ci] bump version of go to 1.13 in github actions config	2019-09-08 19:52:05 +03:00
Aliaksandr Valialkin	9b8af27786	vendor: update github.com/klauspost/compress from v1.7.6 to v1.8.2	2019-09-06 00:49:57 +03:00
Aliaksandr Valialkin	b71d828e84	vendor: update golang.org/x/sys	2019-09-06 00:49:57 +03:00
Aliaksandr Valialkin	1f4e0b722d	deployment: switch docker image tag from v1.27.1-cluster to v1.27.2-cluster	2019-09-05 12:30:03 +03:00
Aliaksandr Valialkin	2c654258ef	lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-05 11:17:17 +03:00
Aliaksandr Valialkin	d0953e9f02	app/vmselect/promql: ignore grouping by destination label in `count_values`, since such a grouping is performed automatically	2019-09-04 19:59:02 +03:00
Aliaksandr Valialkin	2c2bd897dd	lib/storage: remove duplicate tag keys on `MetricName.Marshal` call Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172	2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin	5a9b1d85bb	deployment/docker: switch Go builder from Go 1.12.9 to Go 1.13.0	2019-09-04 17:17:52 +03:00
Aliaksandr Valialkin	f78ffe565f	app/vmselect/promql: do not return artificial points beyond the last point in time series	2019-09-04 16:34:29 +03:00
Aliaksandr Valialkin	a7d5d611fe	app/vmselect/prometheus: do not adjust `start` and `end` args in `/api/v1/query_range` if `nocache=1` arg is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171	2019-09-04 13:10:17 +03:00
Aliaksandr Valialkin	82bfe818d0	lib/fs: try harder with directory removal on NFS in the event of temporary lock Do not give up after 11 attempts of directory removal on laggy NFS. Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts on directory removal. Log failed attempts on directory removal after long sleep times. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-04 12:24:41 +03:00
Alfred Krohmer	7cde25bac4	Don't render extraLabels in ServiceMonitor if they are not given This produced invalid YAML before.	2019-09-03 22:11:24 +03:00
Aliaksandr Valialkin	3182e2a66b	deployment: update docker images from v1.27.0-cluster to v1.27.1-cluster	2019-09-03 21:05:50 +03:00
Aliaksandr Valialkin	b08f085082	app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does	2019-09-03 20:43:29 +03:00
Aliaksandr Valialkin	458d412bb6	app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks The maximum inmemory file size now depends on `-memory.allowedPercent`. This should improve performance and reduce the number of filesystem calls on machines with big amounts of RAM when performing heavy queries over big number of samples and time series.	2019-09-03 13:32:18 +03:00
Aliaksandr Valialkin	0b0153ba3d	lib/storage: invalidate `tagFilters -> TSIDS` cache when newly added index data becomes visible to search Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163	2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin	8504a38214	lib/prombp: apply `ba06b47c16` The following commands used: gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/ gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/	2019-08-29 13:35:54 +03:00
Aliaksandr Valialkin	fb719bfb23	deployment: update docker images from v1.26.0-cluster to v1.27.0-cluster	2019-08-29 00:09:51 +03:00
Aliaksandr Valialkin	8f81908b1f	.github/workflows: added GitHub actions	2019-08-28 23:11:26 +03:00
Aliaksandr Valialkin	604a4312f9	all: port to FreeBSD on GOARCH=amd64	2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin	5893a9f9a3	app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries	2019-08-27 14:28:26 +03:00
Aliaksandr Valialkin	da07a6fb38	lib/storage: go fmt	2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin	a63b69e9e2	lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters	2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin	82e813bad3	lib/storage: properly handle `(?i)` in the tag filter regexp Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin	e2eac858b5	lib/storage: calculate the maximum number of rows per small part from `-memory.allowedPercent` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159 This simplifies error detection additionally to the `vm_rows_ignored_total` counters.	2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin	0a8dd9cc9a	lib/storage: calculate the maximum number of rows per small part from `-memory.allowedPercent` This should improve query speed over recent data on machines with big amounts of RAM	2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin	bc576fb386	lib/storage: properly limit the number of output rows in small and big parts storage Previously small parts storage didn't take into account the available disk space for big parts.	2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin	947decb3dd	lib/storage: remove outdated comment on maxRowsPerSmallPart The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a, which stops merging of small parts on graceful shutdown instead of waiting for their completion.	2019-08-25 13:46:10 +03:00
Artem Navoiev	ce7798a6a2	[deployment] add ingresses and service monitors	2019-08-25 01:04:56 +03:00
Aliaksandr Valialkin	38711526d3	app/vminsert/influx: set `db` label only if Influx line doesnt have `db` tag	2019-08-24 13:55:01 +03:00
Aliaksandr Valialkin	023675c33e	vendor: update github.com/valyala/quicktemplate from v1.1.1 to v1.2.0	2019-08-24 13:39:46 +03:00
Aliaksandr Valialkin	1ee536f9fd	app/vminsert: skip empty tags	2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin	a283023d16	app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric	2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin	38b9615c53	app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric	2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin	2a8fc41bab	app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric	2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin	22685ef94d	app/vminsert/influx: skip invalid rows and continue parsing the remaining rows Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.	2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin	425a81a6c7	app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life The prefious report with escaped newline chars in influx line protocol was false alarm.	2019-08-23 18:43:00 +03:00
Aliaksandr Valialkin	8da8dd0876	app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life	2019-08-23 18:35:52 +03:00
Aliaksandr Valialkin	0ea21eb9dc	app/vminsert/influx: handle `\r\n` aka `crlf` influx line endings from windows world Such lines exist in real life.	2019-08-23 18:28:54 +03:00
Aliaksandr Valialkin	b3502b2b39	app/vminsert/influx: allow escaping newline char Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ , there are reports that such chars occur in real life	2019-08-23 15:14:58 +03:00
Aliaksandr Valialkin	f1f8fce4f7	app/vminsert/influx: skip comments starting with `#` in influx line protocol	2019-08-23 14:43:24 +03:00
Aliaksandr Valialkin	697de90893	app/vminsert: do not drop data in reroutedBuf if all the storage nodes are unhealthy	2019-08-23 10:38:19 +03:00
Aliaksandr Valialkin	a5dc54efc3	app/vminsert: properly limit the size of reroutedBuf	2019-08-23 10:29:51 +03:00
Aliaksandr Valialkin	c50975e12d	vendor: `make vendor-update`	2019-08-23 10:03:42 +03:00
Aliaksandr Valialkin	c197641978	all: return 503 http error if service is temporarily unavailable Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin	e734076f0f	app/vminsert: allow setting the maximum number of labels per time series via `-maxLabelsPerTimeseries`	2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin	4ed63d033a	lib/storage: add benchmarks for regexp filter match / mismatch These benchmarks allow estimate the performance of regexp filters in promql	2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin	559dd03181	deployment: update docker image tags from v1.25.2-cluster to v1.26.0-cluster	2019-08-22 14:58:58 +03:00
Aliaksandr Valialkin	e9db22a551	app/vmselect/promql: attempt to repair invalid bucket counts passed to `histogram_quantile` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154	2019-08-22 14:39:24 +03:00
Aliaksandr Valialkin	0697164b4f	app/vminsert: add ability to ingest data via HTTP OpenTSDB `/api/put` requests This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152 Thanks to nustinov@gmail.com for the initial pull request.	2019-08-22 12:46:54 +03:00
Aliaksandr Valialkin	4d555c7c87	app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing `put` prefixes to each line	2019-08-21 19:15:04 +03:00
Aliaksandr Valialkin	90a4b00b10	app/vmselect/promql: fix panic on `-search.disableCache` Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.	2019-08-21 17:12:01 +03:00
Aliaksandr Valialkin	491b1762c8	app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue	2019-08-21 11:29:41 +03:00
Aliaksandr Valialkin	db1de4277c	app/vmselect/promql: remove NaNs from `/api/v1/query_range` output like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153	2019-08-20 23:01:59 +03:00
Aliaksandr Valialkin	99331606e1	app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries This should reduce memory allocations for big number of timeseries	2019-08-20 23:01:57 +03:00
Aliaksandr Valialkin	1101765adb	app/vmselect/promql: add `label_value(q, label_name)` func, which returns numeric value labels with name `label_name` in `q`	2019-08-20 00:28:44 +03:00
Aliaksandr Valialkin	6ec6a8d7c1	lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with `increase -search.maxUniqueTimeseries` error	2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin	940349ccb9	app/vmselect/promql: independently track offset hints for tStart and tEnd This should improve performance if timeseries starts or ends on the selected time range	2019-08-19 13:40:24 +03:00
Aliaksandr Valialkin	6ae4b4190f	app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do This should improve the performance of queries over big number of time series with big number of output points.	2019-08-19 13:03:38 +03:00
Aliaksandr Valialkin	c59f5c4865	lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders This reduces memory usage and memory fragmentation when working with big number of time series	2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin	45e57be590	deployment: update docker image tags from v1.25.1-cluster to v1.25.2-cluster	2019-08-18 22:56:11 +03:00
Aliaksandr Valialkin	0f45273e20	deployment/docker: switch Go builder from go1.12.8 to go1.12.9	2019-08-18 22:09:21 +03:00
Aliaksandr Valialkin	005aabd305	app/vmselect/promql: add `scrape_interval(q[d])` function, which would return scrape interval for `q` over `d`	2019-08-18 21:08:15 +03:00
Aliaksandr Valialkin	218cb4623a	app/vmselect/promql: hande comparisons with `NaN` similar to Prometheus Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2019-08-18 00:25:58 +03:00
Aliaksandr Valialkin	dcce92c63c	app/vmselect/promql: add `lifetime(q[d])` function, which returns the lifetime of `q` over `d` in seconds. This function is useful for determining time series lifetime. `d` must exceed the expected lifetime of the time series, otherwise the function would return values close to `d`.	2019-08-16 11:59:51 +03:00
Aliaksandr Valialkin	0cb66a8f95	app/vmselect/promql: fix corner-case calculation for `ideriv`	2019-08-16 11:59:50 +03:00
Aliaksandr Valialkin	1b5b9ced27	app/vmselect/promql: properly handle corner cases for rollup functions	2019-08-15 23:31:28 +03:00
Aliaksandr Valialkin	f696cc503a	lib/workingsetcache: automatically detect when it is better to double cache capacity	2019-08-15 22:58:04 +03:00
Aliaksandr Valialkin	97634d7101	deployment/docker: switch Go builder from go1.12.7 to go1.12.8	2019-08-15 20:43:23 +03:00
Aliaksandr Valialkin	e6541a7676	deployment: update docker images	2019-08-15 14:18:43 +03:00
Aliaksandr Valialkin	e399b948de	Makefile: remove duplicate `-cluster` suffix from tar.gz file generated by `make release`, since this suffix must be already present in PKG_TAG	2019-08-15 14:07:43 +03:00
Aliaksandr Valialkin	1dd736a75c	Makefile: add `make release` rule for building release tar.gz file with cluster binaries Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/146	2019-08-15 14:05:56 +03:00
Aliaksandr Valialkin	c15dfc6cea	vendor: update github.com/valyala/gozstd from v1.5.1 to v1.6.0	2019-08-15 12:57:59 +03:00
Aliaksandr Valialkin	83ed5d3109	deployment: update docker images	2019-08-14 03:12:49 +03:00
Aliaksandr Valialkin	99eed2ca14	lib/storage: properly cache tagFilters -> TSIDs entries from historical index	2019-08-14 02:32:25 +03:00
Aliaksandr Valialkin	f1d81b9405	lib/storage: compress contents of cache for tagFilters -> TSIDs This should increase cache capacity	2019-08-14 02:32:22 +03:00
Aliaksandr Valialkin	b8bbe92de1	app/vmselect/promql: store compressed results in the cache This should increase rollup results cache capacity.	2019-08-14 02:32:16 +03:00
Aliaksandr Valialkin	8c2158af24	all: use workingsetcache instead of fastcache This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin	51263b1a45	lib/fs: add test for IsTemporaryFileName	2019-08-13 21:33:54 +03:00
Aliaksandr Valialkin	867612a4a4	Makefile: consistency renaming: check_all -> check-all	2019-08-13 21:32:08 +03:00
Aliaksandr Valialkin	5a7ab0d90b	lib/storage: remove broken BenchmarkIndexDBSearchTSIDs	2019-08-13 20:21:23 +03:00
Aliaksandr Valialkin	39f3f3a517	lib: move common code for creating flock.lock file into fs.CreateFlockFile	2019-08-13 01:46:20 +03:00
Aliaksandr Valialkin	73f866d874	lib/fs: atomically create file with the given contents on WriteFileAtomically This should prevent from `transaction` and `metadata.json` files corruption on unclean shutdown such as OOM, `kill -9`, power loss, etc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148	2019-08-12 15:02:04 +03:00
Aliaksandr Valialkin	ad5be625f8	deployment: update docker images	2019-08-06 16:10:03 +03:00
Aliaksandr Valialkin	4fb635b0c9	lib/storage: do not change timestamps to constant rate if values are constant or have constant delta This breaks the original timestamps, which results in issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .	2019-08-06 15:40:17 +03:00
Aliaksandr Valialkin	f56c1298ad	app/vmstorage: add `vm_concurrent_addrows_*` metrics for tracking concurrency for Storage.AddRows calls Track also the number of dropped rows due to the exceeded timeout on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`	2019-08-06 15:08:43 +03:00
Aliaksandr Valialkin	2d869c6d9b	vendor: update github.com/VictoriaMetrics/metrics to v1.7.1	2019-08-05 19:21:53 +03:00
Aliaksandr Valialkin	8e05758ff5	app: add `vm_concurrent_` metrics for visibility in concurrency limiters for vminsert and vmselect	2019-08-05 18:30:29 +03:00
Aliaksandr Valialkin	1258c9ef10	vendor: `make vendor-update`	2019-08-05 10:34:38 +03:00
Aliaksandr Valialkin	a3ecf3c1f7	lib/storage: properly reset `partSearch.fetchData` in `partSearch.reset`	2019-08-05 09:55:50 +03:00
Artem Navoiev	dd4ea63ed2	[deployment] add statefulset for vmselect (#140 )	2019-08-04 23:34:05 +03:00
Aliaksandr Valialkin	a868f8607f	deployment: update docker images to v1.24.0-cluster	2019-08-04 23:31:57 +03:00
Aliaksandr Valialkin	53c8f56436	app/vmselect: allow passing `match[]`, `start` and `time` to `/api/v1/label/<label_name>/values` `/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)` call in Grafana templating.	2019-08-04 23:07:00 +03:00
Aliaksandr Valialkin	880b1d80b1	app/vmselect: optimize `/api/v1/series` by skipping storage data Fetch and process only time series metainfo.	2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin	7f5afae1e3	app/vmselect/prometheus: prevent from fetching and scanning all the data on `/api/v1/searies` call by default	2019-08-04 19:42:45 +03:00
Aliaksandr Valialkin	000c154641	app/vmselect/promql: tune automatic window adjustement Increase the windows adjustement for small scrape intervals, since they usually have higher jitter. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134	2019-08-04 19:34:11 +03:00
Aliaksandr Valialkin	1d4ddadbb1	app/vmselect/promql: further increase the allowed jitter for scrape interval Real-world production data shows higher jitter than 1/8 of scrape interval. This may results in gaps on the graph. So increase the allowed jitter to 1/4 of scrape interval in order to reduce the probability of gaps on the graphs over time series with high jitter for scrape_interval.	2019-08-02 20:16:41 +03:00
Aliaksandr Valialkin	8ed84a4713	app/vminsert/influx: round automatically generated timestamp according to the given `precision` arg	2019-08-02 00:24:39 +03:00
Aliaksandr Valialkin	ade7bc30db	app/vmselect/promql: tolerate higher jitter in scrape interval Allow jitter for up to 1/8 instead of 1/16 for the scrape interval. This should imrpove graphs when `step` is smaller than the `scrape_interval`.	2019-08-01 23:25:53 +03:00
Aliaksandr Valialkin	a99e89945e	lib/decimal: modernize tests a bit	2019-07-31 21:09:54 +03:00
Aliaksandr Valialkin	6fceedccce	deployment: update docker images	2019-07-31 16:38:39 +03:00
Aliaksandr Valialkin	c994fbf500	app/vmselect/promql: add `vm_slow_queries_total` metric for counting slow queries The query is slow if its execution time exceeds `-search.logSlowQueryDuration`	2019-07-31 03:36:45 +03:00
Aliaksandr Valialkin	071a122119	app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken	2019-07-31 01:18:11 +03:00
Aliaksandr Valialkin	b9a16b93e7	app/vmselect/promql: allow adjusting window for default rollup function Default rollup function is `last_over_time`. It must support adjusting the provided window in order to prevent from gaps on the graph for window values smaller than scrape interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134	2019-07-31 00:45:58 +03:00
Aliaksandr Valialkin	c901a6472f	app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136	2019-07-30 22:05:55 +03:00
Aliaksandr Valialkin	b7c4b0c6d2	lib/storage: fix matching against tag filter with empty name Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137	2019-07-30 15:15:21 +03:00
Aliaksandr Valialkin	5b8526e925	app/vmselect/netstorage: improve error message when reading data blocks from storage Mention the block number in the error. This should simplify troubleshooting in this code.	2019-07-28 12:17:33 +03:00
Aliaksandr Valialkin	b7089705b7	app/vminsert: add `vm_rows_per_insert` summary metric This metric should help tuning batch sizes on clients writing data to VictoriaMetrics	2019-07-27 13:28:20 +03:00
Aliaksandr Valialkin	1fd4e9fb5c	app/vminsert: improve error messages for Influx, OpenTSDB and Graphite parsing Include in the error message the line which failed to parse. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/127	2019-07-26 22:09:21 +03:00
Aliaksandr Valialkin	34b21a8671	deployment: update cluster image to v1.23.0-cluster	2019-07-26 20:07:47 +03:00
Aliaksandr Valialkin	8253790157	app/vmstorage: consistency renaming for `ignored rows` metrics vm_too_big_timestamp_rows_total -> vm_rows_ignored_total{reason="big_timestamp"} vm_too_small_timestamp_rows_total -> vm_rows_ignored_total{reason="small_timestamp"}	2019-07-26 20:02:24 +03:00
Aliaksandr Valialkin	c6bec48927	lib/storage: add metrics for calculating skipped rows outside the retention The metrics are: - vm_too_big_timestamp_rows_total - vm_too_small_timestamp_rows_total	2019-07-26 14:11:56 +03:00
Aliaksandr Valialkin	aac482517f	app/vmselect/promql: return NaN from `count()` over zero time series This aligns `count` behavior with Prometheus.	2019-07-25 22:02:34 +03:00
Aliaksandr Valialkin	0e52357f35	app/vmselect/promql: properly calculate incremental aggregations grouped by `__name__` Previously the following query may fail on multiple distinct metric names match: sum(count_over_time{__name__!=''}) by (__name__)	2019-07-25 21:53:26 +03:00
Aliaksandr Valialkin	f2e8d54fb0	lib/encoding/zstd: `go fmt`	2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin	97b5dc7122	lib/encoding/zstd: disable CRC checks in `pure Go` build This should give slightly better compression and decompressions performance. Additionally this shaves off 4 bytes per each compressed block.	2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin	54f035d4ce	all: small updates after PR #114	2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin	7a133567fb	lib/encoding: small fixes in tests after the PR #114	2019-07-24 17:43:39 +03:00
Roman Khavronenko	fcf09aaa3c	all: add Pure Go build (pull request #114 ) Updates #94	2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin	dd7bba94a3	dashboards: use `rate` instead of `irate`, because `irate` doesn't capture spikes See https://medium.com/@valyala/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832 for details	2019-07-20 15:55:48 +03:00
Aliaksandr Valialkin	3fae34eeb4	lib/encoding: improve gauge series detection - Series with negative values are always gauges - Counters may only have increasing values with possible counter resets This should improve compression ratio for gauge series which were previously mistakenly detected as counters.	2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin	b335a811c3	deployment: switch builder from go1.12.6 to go1.12.7	2019-07-20 12:14:05 +03:00
Jiri Tyr	0aed0e0b5d	Adding Grafana dashboards for VM cluster (#105 )	2019-07-20 10:25:09 +03:00
Aliaksandr Valialkin	cb8104cf77	app: clarify error messages when `-storageNode` arg is missing in vminsert and vmselect	2019-07-20 10:21:59 +03:00
Aliaksandr Valialkin	fab1962e02	deployment/k8s/helm: use correct default ports for -storageNode Previously these ports were swapped. Correct ports are: - vminsert: -storageNode=:8400 - vmselect: -storageNode=:8401	2019-07-20 01:24:32 +03:00
Aliaksandr Valialkin	e3dcfe5851	deployment/docker/docker-compose.yml: use default ports for `vminsert` and `vmselect` services These ports were swapped. Correct default ports are: - vminsert: -httpListenAddr=:8480, -storageNode=:8400 - vmselect: -httpListenAddr=:8481, -storageNode=:8401	2019-07-20 01:20:08 +03:00
Thor Anker Kvisgård Lange	f576b267eb	Fixed small bug in vmstorage name template Signed-off-by: Thor Anker Kvisgård Lange <thanl@mhivestasoffshore.com>	2019-07-17 13:30:23 +03:00
Aliaksandr Valialkin	76b947dcb4	deployment: update Docker images	2019-07-15 23:56:24 +03:00
Aliaksandr Valialkin	7abb96b454	lib/netutil: do not count timeouts as network errors	2019-07-15 23:06:13 +03:00
Aliaksandr Valialkin	2b4254d01f	app/vminsert: use netutil.TCPListener for collecting network-related metrics for Graphite and OpenTSDB TCP traffic	2019-07-15 22:58:35 +03:00
Aliaksandr Valialkin	092c9b39a8	app/vmselect/promql: remove empty time series after applying filters like `q > 0` This should reduce CPU and RAM usage for queries over high number of time series.	2019-07-12 19:59:49 +03:00
Aliaksandr Valialkin	3bc9d3a14c	vendor: update github.com/VictoriaMetrics/metrics to v1.7.0 This version adds support for `process_*` metrics similar to metrics exposed by https://github.com/prometheus/client_golang . Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/92	2019-07-12 17:24:58 +03:00
Aliaksandr Valialkin	6875fb411a	app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores This may reduce response times for aggregation over big number of time series with small step between output data points.	2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin	be0ce54010	deployment: update docker images	2019-07-12 02:35:09 +03:00
Aliaksandr Valialkin	73a47d2a53	lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs	2019-07-12 02:28:40 +03:00
Aliaksandr Valialkin	97f9397687	lib/storage: do not reduce `maxMetrics` on time ranges exceeding `maxDaysForDateMetricIDs` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95	2019-07-12 02:21:52 +03:00
Aliaksandr Valialkin	1de6ef5f51	deployment: update Docker images	2019-07-11 19:10:35 +03:00
Aliaksandr Valialkin	4a8e6f47fe	app/vmselect/prometheus: set `start` arg in `/api/v1/series` to the minimum allowed time by default as Prometheus does Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91	2019-07-11 17:11:37 +03:00
Aliaksandr Valialkin	3313cdf816	app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage	2019-07-11 17:11:35 +03:00
Aliaksandr Valialkin	4ca66344ee	lib/storage: do not pollute inverted index with data for samples outside the retention period	2019-07-11 17:11:33 +03:00
Aliaksandr Valialkin	0522efb2d6	lib/storage: add missing tagFilter.Marshal func	2019-07-11 15:01:01 +03:00
Aliaksandr Valialkin	12b1d67b41	lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value	2019-07-11 14:48:51 +03:00
Aliaksandr Valialkin	bf2e1b0ac1	lib/storage: remember and skip individual tag filters matching too many metrics This saves CPU time by skipping useless matching for individual tag filters.	2019-07-11 14:48:47 +03:00
Aliaksandr Valialkin	cbab86fd9d	app/vmselect/promql: reduce RAM usage for aggregates over big number of time series Calculate incremental aggregates for `aggr(metric_selector)` function instead of keeping all the time series matching the given `metric_selector` in memory.	2019-07-10 13:03:36 +03:00
Aliaksandr Valialkin	ba8195c58e	all: consistency renaming: bytesSize -> sizeBytes	2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin	df6f17b82c	app/vmselect/promql: mention `-search.logSlowQueryDuration` flag value in the slow query log message	2019-07-10 00:43:01 +03:00
Aliaksandr Valialkin	73ae889244	app/vmselect/promql: extract `rmoeveGroupTags` function for removing unneeded tags from MetricName according to the given modifierExpr	2019-07-09 23:20:58 +03:00
Aliaksandr Valialkin	603b34edbd	app/vmselect/promql: properly preserve metric name after applying functions in any case from `transformFuncsKeepMetricGroup`	2019-07-09 23:10:49 +03:00
Aliaksandr Valialkin	d6ec95693d	app/vmselect/prometheus: typo fix	2019-07-07 23:34:04 +03:00
Aliaksandr Valialkin	61f6f63964	README.md: mention that `vminsert` spreads data amount `vmstorage` nodes using consistent hashing	2019-07-07 22:00:45 +03:00
Aliaksandr Valialkin	36636c1f6f	app/vmselect/prometheus: handle `minTime` and `maxTime` values that may be set by Promxy or Prometheus client Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/88	2019-07-07 21:53:52 +03:00
Aliaksandr Valialkin	50c5894dc0	deployment: update docker images	2019-07-04 19:54:27 +03:00
Aliaksandr Valialkin	bba07d05fe	app/vmselect/promql: remove empty timeseries left after `topk` call	2019-07-04 19:43:07 +03:00
Aliaksandr Valialkin	41f512af1c	all: add `vm_data_size_bytes` metrics for easy monitoring of on-disk data size and on-disk inverted index size	2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin	512a627855	app/vmselect/prometheus: update adjustLastPoints function - Do not overwrite last points by the previous NaNs, since this may result in empty time series. - Overwrite the last 2 points instead of 3. This should be enough in most cases.	2019-07-04 09:30:56 +03:00
Aliaksandr Valialkin	858746fa6c	app/vmselect/promql: gracefully handle duplicate timestamps in `irate` and `rollup_rate` funcs Previously such timestamps result in `+Inf` results. Now the previous timestamp is used for the calculations.	2019-07-03 12:41:30 +03:00
Aliaksandr Valialkin	81da1c7b47	README.md: stylistic updates	2019-07-02 15:57:45 +03:00
Aliaksandr Valialkin	a3abed80ff	app/vmselect: do not return empty time series in `/api/v1/query` result	2019-07-01 17:16:26 +03:00
Aliaksandr Valialkin	6682a35731	lib/memory: attempt #3 to determine memory limit for LXC container Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-07-01 14:01:57 +03:00
Aliaksandr Valialkin	c3c60bee45	app/vmselect: add `-search.denyPartialResponse` flag for disabling partial responses if some of vmstorage nodes are unavailable Also accept `deny_partial_response` query arg in Prometheus API handlers. If it is set to true, then return error if some of vmstorage nodes are unavailable.	2019-06-30 01:27:07 +03:00
Aliaksandr Valialkin	60cff62586	deployment: update docker images	2019-06-29 21:25:20 +03:00
Aliaksandr Valialkin	b6ea1a7d5e	lib/mergeset: `make fmt`	2019-06-29 14:25:46 +03:00
Aliaksandr Valialkin	ffc1bb00f6	lib/storage: skip non-matching metricIDs in sortedFilter This should improve performance for big sorteFilter lists.	2019-06-29 13:49:40 +03:00
Aliaksandr Valialkin	2257dcd278	lib/mergeset: speed up binarySearchKey by skipping the first item during binary search	2019-06-29 13:49:32 +03:00
Aliaksandr Valialkin	72a3050c41	app/vmselect/promql: consistency renaming: candlestick -> rollup_candlestick	2019-06-29 03:13:25 +03:00
Aliaksandr Valialkin	6ea12a079e	lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis	2019-06-28 18:23:06 +03:00
Aliaksandr Valialkin	d0732d3137	lib/memory: attempt #2 to determine memory limit inside LXC container Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 18:08:57 +03:00
Aliaksandr Valialkin	628571a837	README.md: update cluster scheme	2019-06-28 17:54:13 +03:00
Aliaksandr Valialkin	ad436757c3	lib/memory: an attempt to read proper memory limit inside LXC container Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 15:35:02 +03:00
Aliaksandr Valialkin	c6598a8507	vendor: update github.com/VictoriaMetrics/metrics to v1.6.2 This fixes Summary printing for _count and _sum values with metric names containing labels.	2019-06-28 14:18:24 +03:00
Aliaksandr Valialkin	4f8cbc0782	vendor: update github.com/VictoriaMetrics/metrics to v1.6.1	2019-06-28 14:06:32 +03:00
Aliaksandr Valialkin	391bc8bf38	app/vmselect: fix 32bit arm build Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/83	2019-06-27 19:37:17 +03:00
Aliaksandr Valialkin	2d497c3b8e	README.md: add cluster scheme image	2019-06-27 19:23:36 +03:00
Aliaksandr Valialkin	96342f1422	app/vmselect: add `candlestick(m[d])` func for returning `open`, `close`, `low` and `high` rollups on the given time range `d` This function is frequently used in financial apps. See https://en.wikipedia.org/wiki/Candlestick_chart	2019-06-27 18:46:54 +03:00
Aliaksandr Valialkin	416d27ef11	lib/storage: optimize time series search by regexp filter This should improve search speed on label filters like `{foo=~"bar.+baz"}`	2019-06-27 16:18:00 +03:00
Aliaksandr Valialkin	5850a9ea78	README.md: improve wording on gossip protocol	2019-06-26 23:50:34 +03:00
Aliaksandr Valialkin	05b7cb1d42	README.md: return back the link to unsuccessful attempt to implement Gossip protocol in Thanos This link provides good information on gossip fragility	2019-06-26 23:48:48 +03:00
Jiri Tyr	e7a0bf1a71	Change the default influxMeasurementFieldSeparator	2019-06-26 13:22:54 +03:00
Aliaksandr Valialkin	d5cb9fddd8	app/vminsert: fix inifinite loop when reading two lines without newline in the end Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/82	2019-06-26 02:52:56 +03:00
Aliaksandr Valialkin	916d9ef5b3	deployment: update docker images	2019-06-25 21:49:23 +03:00
Aliaksandr Valialkin	4f54bcf90b	app/vmselect/promql: suppress error when template func is used inside modifier list. Just leave it as is Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/78	2019-06-25 20:43:57 +03:00
Aliaksandr Valialkin	72873f67aa	README.md: improve docs for Helm chapter	2019-06-25 20:13:47 +03:00
Aliaksandr Valialkin	ee23a143b9	lib/storage: make sure non-nil args are passed to openIndexDB	2019-06-25 20:10:08 +03:00
Aliaksandr Valialkin	8b0a63722f	lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive This should improve performance on inverted index search for big amount of unique time series when big -search.maxUniqueTimeseries is set.	2019-06-25 19:57:31 +03:00
Aliaksandr Valialkin	0263cb0adc	lib/storage: free up memory from caches owned by indexDB when it is deleted	2019-06-25 14:41:16 +03:00
Aliaksandr Valialkin	362e187011	lib/storage: use unversioned keys for tag cache in extDB Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache. This should improve performance for index lookups over big amount of time series.	2019-06-25 13:15:42 +03:00
Aliaksandr Valialkin	51e2f3b48f	lib/storage: skip searching in extDB if it doesn't contain items for the given time range This should improve inverted index search performance for big amount of unique time series when the search is performed only on recent data.	2019-06-25 12:57:56 +03:00
Aliaksandr Valialkin	dbc1e87bac	deployment: update docker images	2019-06-24 23:11:03 +03:00
Aliaksandr Valialkin	d0bf4393a9	app/vmselect/promql: increase default value for -search.maxPointsPerTimeSeries from 10k to 30k This may be required for subqueries with small steps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/77	2019-06-24 22:53:25 +03:00
Aliaksandr Valialkin	334cf253c7	app/vmselect/promql: adjust value returned by linearRegression to the end of time range like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71	2019-06-24 22:46:03 +03:00
Aliaksandr Valialkin	14cd628948	app/vmselect/promql: add `sum2` and `sum2_over_time`, `geomean` and `geomean_over_time` funcs. These functions may be useful for statistic calculations.	2019-06-24 16:45:00 +03:00
Aliaksandr Valialkin	fb9358635d	lib/storage: mention source parts on merge error This should improve determining broken source part. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76	2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin	0eac538fc8	app/vmselect/promql: adjust the provided window only for range functions with dt in denominator This should fix range function calculations such as `changes(m[d])` where `d` is smaller than the scrape interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72	2019-06-23 19:27:25 +03:00
Aliaksandr Valialkin	ec57e59154	app/vmselect/promql: use `deriv_fast` instead of `deriv` in `ttf`, since `deriv` calculations have been changed recently	2019-06-23 15:54:12 +03:00
Aliaksandr Valialkin	516062b162	app/vmselect/promql: adjust `ttf` calculation, so `deriv(freev)` for `freev`=`m[d]` could be properly calculated	2019-06-23 14:31:36 +03:00
Aliaksandr Valialkin	5ea5ec4f44	vendor: update github.com/valyala/gozstd to v1.5.1	2019-06-22 00:14:11 +03:00
Aliaksandr Valialkin	ef6ca22c1d	deployment: update docker images	2019-06-21 23:35:48 +03:00
Aliaksandr Valialkin	a4e040f5ef	app/vmselect/promql: typo fixes in comments	2019-06-21 23:22:54 +03:00
Aliaksandr Valialkin	c05d443791	app/vmselect/promql: add `deriv_fast` function for calculating fast derivative `deriv_fast` calculates derivative based on the first and the last point on the interval instead of calculating linear regression based on all the data points on the interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73	2019-06-21 23:05:48 +03:00
Aliaksandr Valialkin	98eafdbd58	app/vmselect/promql: use linear regression in `deriv` func like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73	2019-06-21 22:54:34 +03:00
Aliaksandr Valialkin	f334908c22	app/vmselect/promql: ajdust data model to the model used in Prometheus Do not take into account data points on the range `[timestamp .. timestamp+step)` when calculating value on the given `timestamp`. Use only data points from the past when performing these calculations like Prometheus does. This should reduce discrepancies between results returned by VictoriaMetrics and results returned by Prometheus. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71	2019-06-21 21:55:25 +03:00
Aliaksandr Valialkin	0fc4cb67dc	deployment: update docker images	2019-06-21 13:39:45 +03:00
Aliaksandr Valialkin	837e349b7d	app/vmselect/promql: do not strip `__name__` form time series after binary comparison operation Example: foo > 10 Would leave `foo` name for all the matching time series on the left.	2019-06-21 13:08:02 +03:00
Aliaksandr Valialkin	9164c223ec	all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2019-06-20 20:07:41 +03:00
Aliaksandr Valialkin	786beb8fc8	Makefile: enable golangci-lint in `make check_all`	2019-06-20 15:00:58 +03:00
Aliaksandr Valialkin	9cac11db64	lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:38:45 +03:00
Aliaksandr Valialkin	7778030f9f	lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:36:19 +03:00
Aliaksandr Valialkin	e84b7641ef	app/vmselect/promql: remove unused func `keepLastValue`; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:35:19 +03:00
Aliaksandr Valialkin	db042bf6d6	app/vmselect/promql: typo fix; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:33:52 +03:00
Aliaksandr Valialkin	dec2bdf89f	Makefile: add `make golangci-lint` rule for running `golangci-lint run`; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:32:34 +03:00
Aliaksandr Valialkin	3838d224d5	app/vminsert/opentsdb: remove unused const `maxReadPacketSize`; update https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:30:02 +03:00
Aliaksandr Valialkin	a3a53647ba	app/vmselect/prometheus: return better error messages on missing args to `/api/v1/*`	2019-06-20 14:07:44 +03:00
Aliaksandr Valialkin	a0c22a6830	app/vmstorage: add `vm_cache_entries{type="storage/hour_metric_ids"}` metric for tracking active time series count	2019-06-19 18:37:38 +03:00
Aliaksandr Valialkin	08e255a206	README.md: add link to source codes for cluster branch	2019-06-19 17:56:56 +03:00
Aliaksandr Valialkin	24ae3ef532	lib/prompb: remove superflouos bytes copying in ReadSnappy	2019-06-18 21:02:02 +03:00
Aliaksandr Valialkin	d4ed6189d4	app/vminsert/graphite: allow skipping timestamps in Graphite plaintext protocol In this case VictoriaMetrics uses the ingestion time as a timestamp.	2019-06-18 19:05:46 +03:00
Aliaksandr Valialkin	7b93da5b57	vendor: update golang.org/x/sys	2019-06-18 16:20:09 +03:00
Aliaksandr Valialkin	2ebcd0c98b	deployment: update docker images	2019-06-18 13:36:42 +03:00
Aliaksandr Valialkin	e40224d5de	lib/flagutil: add NewArray helper func	2019-06-18 10:44:09 +03:00
Aliaksandr Valialkin	02417071cd	README.md: use link to Wikipedia about broken gossip protocol instead of a link to document about Gossip protocol removal from Thanos Thanos removed non-working gossip protocol a few months ago - https://github.com/improbable-eng/thanos/issues/734 , so the link to the design document https://github.com/improbable-eng/thanos/blob/master/docs/proposals/approved/201809_gossip-removal.md became unavailable. So use a link to Wikipedia article instead. Closes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/68	2019-06-17 19:10:29 +03:00
Aliaksandr Valialkin	3b16d49514	app/vminsert/influx: add `-influxSkipSingleField` flag for using `{measurement}` instead of `{measurement}{separator}{field_name}` for Influx lines with a single field Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/66	2019-06-17 19:05:46 +03:00
Aliaksandr Valialkin	5f0b3589b2	app/vminsert/influx: add -influxMeasurementFieldSeparator flag for the ability to change separator for `{measurement}{separator}{field_name}` metric name	2019-06-14 09:57:13 +03:00
Aliaksandr Valialkin	14edd122a6	deployment/docker: switch builder from go1.12.5 to go1.12.6	2019-06-14 09:31:56 +03:00
Aliaksandr Valialkin	f9e1d32168	lib/storage: persist metric ids for the current and the previous hour on graceful shutdown This should improve performance after restart when the db contains a lot of time series with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)	2019-06-14 07:55:09 +03:00
Aliaksandr Valialkin	ba3cccd471	deployment: update docker images	2019-06-12 23:31:06 +03:00
Aliaksandr Valialkin	947bc16f8c	app/vmselect/promql: use dynamic limit on memory for concurrent queries	2019-06-12 23:18:23 +03:00
Aliaksandr Valialkin	fe1b33ef1a	README.md: mention that accountID is known as tenant	2019-06-12 21:32:10 +03:00
Aliaksandr Valialkin	8567e3463d	app/vmselect/promql: merge non-overlapping duplicate time series in `group_left` and `group_right` joins	2019-06-12 20:33:01 +03:00
Aliaksandr Valialkin	345ecc37b6	deployment: update docker images	2019-06-12 18:36:17 +03:00
Aliaksandr Valialkin	88005237f4	app/vmselect/promql: swap binary operation with modifier in the error message for improved readability	2019-06-12 17:14:33 +03:00
Aliaksandr Valialkin	a71381ad2a	app/vmselect/promql: list a sample of duplicate time series in the error message for `group_left` or `group_right` This should improve troubleshooting for complex queries involving `group_left` and `group_right` modifiers.	2019-06-12 16:57:34 +03:00
Aliaksandr Valialkin	b0b93e3d50	lib/fs: sync parent dir in MustRemoveAll only if it exists The parent directory may be non-existing when the deleted directory didn't exist before the MustRemoveAll call	2019-06-12 02:16:15 +03:00
Aliaksandr Valialkin	18d6f293f7	lib/fs: consolidate RemoveAll funcs into a single MustRemoveAll func The func syncs parent dir in order to persist directory removal in the event of power loss	2019-06-12 01:55:18 +03:00
Aliaksandr Valialkin	28d9904efc	lib/fs: panic with fatal error when directories cannot be removed Unremoved directories may lead to inconsistent data directory, so VictoriaMetrics will fail to start next time. So panic on the first error when trying to remove directory in order to simplify recover process.	2019-06-12 01:20:10 +03:00
Aliaksandr Valialkin	d897bc3f08	lib/fs: attempt #2 to work around NFS issue with directory removal Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-12 01:07:29 +03:00
Aliaksandr Valialkin	f165500225	vendor: update github.com/VictoriaMetrics/fastcache to v1.5.1	2019-06-11 23:57:15 +03:00
Aliaksandr Valialkin	d1ca2e5a2d	deployment: update docker images	2019-06-11 23:22:51 +03:00
Aliaksandr Valialkin	51e2e255a6	lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error	2019-06-11 23:13:45 +03:00
Aliaksandr Valialkin	3fa4c28f6b	lib/fs: make sure the created directory remains visible in the fs in the event of power loss Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 23:08:17 +03:00
Aliaksandr Valialkin	0b7f751f60	lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused	2019-06-11 22:54:23 +03:00
Aliaksandr Valialkin	cb9e746484	deployment: update docker images	2019-06-11 22:02:08 +03:00
Aliaksandr Valialkin	b491045a4b	lib/{storage,mergeset}: sync filenames inside part when finalizing the part Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 21:51:19 +03:00
Aliaksandr Valialkin	3437c30180	all: try hard removing directory with contents Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-11 01:58:08 +03:00
Aliaksandr Valialkin	f2a8599908	deployment: update docker images	2019-06-11 01:09:52 +03:00
Aliaksandr Valialkin	eea7da8e0c	app/vmselect/promql: prevent from `count_values` explosion of timeseries, which could result in OOM	2019-06-11 01:03:18 +03:00
Aliaksandr Valialkin	e87a602209	app/vmselect/promql: skip superflouos timestamps copying in `count_values`	2019-06-11 00:44:09 +03:00
Aliaksandr Valialkin	ec84febc1c	app/vmselect/promql: remove superflouos timeseries copy in `histogram_quantile` func	2019-06-11 00:39:35 +03:00
Aliaksandr Valialkin	1fab34fb5c	app/vmselect/promql: remove superflouos timeseries copy in `union` func	2019-06-11 00:35:09 +03:00
Aliaksandr Valialkin	a6f368499d	app/vmselect/promql: skip NaN values in `count_values` func	2019-06-10 22:42:41 +03:00
Aliaksandr Valialkin	2d7165033a	deployment: update docker images	2019-06-10 20:38:18 +03:00
Aliaksandr Valialkin	945894e049	app/vmselect: properly handle empty label (aka __name__) in LabelEntries handler	2019-06-10 19:55:02 +03:00
Aliaksandr Valialkin	75a0acf72d	app/vmselect: add `/api/v1/labels/count` handler for quick detection of labels with the maximum number of distinct values	2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin	547bcdce63	app/vmstorage: enable compression of responses to vmselect by default This should save vmstorage => vmselect network bandwidth in common case when recently added data is queried.	2019-06-10 14:54:59 +03:00
Aliaksandr Valialkin	0ccedbdfd2	lib/storage: mention the accountID and projectID in error message when filtering out other (accountID, projectID) entries	2019-06-10 14:43:53 +03:00
Aliaksandr Valialkin	d54f5fec0b	lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter This should improve speed for searching metrics among high number of time series with high churn rate like in big Kubernetes clusters with frequent deployments.	2019-06-10 14:07:47 +03:00
Aliaksandr Valialkin	27e50e86f4	lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters	2019-06-10 13:26:00 +03:00
Aliaksandr Valialkin	b69d3dbd0c	lib/storage: filter out metricIDs from another (AccountID, ProjectID) in getMetricIDsForRecentHours	2019-06-10 13:05:16 +03:00
Aliaksandr Valialkin	3059ae7be0	lib/storage: give clearer names to more functions	2019-06-10 12:59:33 +03:00
Aliaksandr Valialkin	d3a024d2d6	lib/storage: give more clear names to functions	2019-06-10 12:50:22 +03:00
Aliaksandr Valialkin	00e0760608	lib/storage: test GetSeriesCount	2019-06-10 12:40:33 +03:00
Aliaksandr Valialkin	e4cba5a7ed	lib/storage: make getSeriesCount func indexSearch method	2019-06-10 12:29:24 +03:00
Aliaksandr Valialkin	4c3913290a	app/vmstorage: add missing `_total` suffixes to newly added metrics	2019-06-09 22:11:41 +03:00
Aliaksandr Valialkin	d882afa905	lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)	2019-06-09 19:14:04 +03:00
Aliaksandr Valialkin	5fcdb4a59a	app/vminsert: improve handling of unhealthy vmstorage nodes * Spread load evenly among remaining healthy nodes instead of hammering the next node after the unhealthy node. * Make sure that the packet is flushed to storage node before returning success. Previously packets could stay in local buffers and thus lost on connection errors. * Keep rows in the limited memory when all the storage nodes are unhealthy.	2019-06-09 00:42:36 +03:00
Aliaksandr Valialkin	0f64673327	app/vminsert/concurrencylimiter: typo fix in the error message	2019-06-08 22:43:56 +03:00
Aliaksandr Valialkin	89a113cb5d	app/vminsert: really fix #60 ReadLinesBlock may accept dstBuf with non-zero length. In this case the last line without trailing newline isn't read. Fix this by comparing len(dstBuf) to 0 instead of its original length.	2019-06-07 23:40:10 +03:00
Aliaksandr Valialkin	e1c45b314a	app/vminsert: properly read trailing line without newline in the end This fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/60	2019-06-07 23:18:34 +03:00
Aliaksandr Valialkin	8cf0a0e59c	app/vminsert: split vm_rows_inserted_total into per-(accountID, projectID) metrics Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/59	2019-06-07 22:11:20 +03:00
Aliaksandr Valialkin	8b2a6c6182	deployment: update docker images	2019-06-07 11:58:33 +03:00
Aliaksandr Valialkin	30c7652bad	deployment/docker: move cluster docker images from `valyala/vm` to `victoriametrics/vm` docker hub path	2019-06-07 11:55:37 +03:00
Aliaksandr Valialkin	41d087662c	deployment: update docker image	2019-06-07 11:40:54 +03:00
Aliaksandr Valialkin	913f888d0c	app/vmselect/promql: properly handle `{__name__ op "string"}` queries This has been broken in 7294ef333ad26f4f6578b783e97649e58b1f8945 .	2019-06-07 02:02:09 +03:00
Seua Polyakov	5e51ce386e	Add SIGINT as stopsignal to docker file (#54 ) Add sigint as stopsignal to docker file. You can find more here: https://docs.docker.com/engine/reference/builder/#usage With this change, the main process inside the container will receive SIGINT, and after a grace period, SIGKILL. (cherry picked from commit `f4e63cd070`)	2019-06-06 22:38:48 +03:00
Aliaksandr Valialkin	11979e4d85	app/vmselect/prometheus: report about incorrect time or duration instead of silently using the default value This should prevent from incorrect usage of the querying API. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/52	2019-06-06 22:17:15 +03:00
Aliaksandr Valialkin	5f2aa4539a	app/vminsert: add multi-tenancy support for OpenTSDB and Graphite ingestion via custom tags * VictoriaMetrics_AccountID tag may be used for setting AccountID * VictoriaMetrics_ProjectID tag may be used for setting ProjectID	2019-06-06 18:07:30 +03:00
Aliaksandr Valialkin	c98582695f	deployment: update docker images	2019-06-06 17:39:31 +03:00
Aliaksandr Valialkin	8f4790625d	app/vmselect/promql: return the correct time series from `quantile` Previously arbitrary time series could be returned from `quantile` depending on sort order for the last data point in the selected range. Fix this by returning the calculated time series. Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/55	2019-06-06 17:33:53 +03:00
Aliaksandr Valialkin	2ff0d595b0	app/vmselect/promql: add `-search.disableCache` flag for disabling response caching This may be useful for data back-filling, when the response caching could interfere badly with newly added data points with timestamps in the past.	2019-06-04 17:30:41 +03:00
Aliaksandr Valialkin	595a421295	deployment: update docker images	2019-06-03 23:00:10 +03:00
Aliaksandr Valialkin	ba58af9d8c	app/vminsert/influx: take into account all the tags for consistent hash calculations	2019-06-03 22:54:21 +03:00
Aliaksandr Valialkin	db21d46417	app/vminsert: emulate influx/query request, which is required for TSBS benchmark	2019-06-03 18:39:46 +03:00
Aliaksandr Valialkin	8ad0fb5689	deployment: update docker images	2019-06-03 18:21:18 +03:00
Aliaksandr Valialkin	31d6566aff	app/vminsert: accept data on `/insert/<accountID>/prometheus/api/v1/write`	2019-06-03 18:18:09 +03:00
Aliaksandr Valialkin	c3d73e347c	deployment/k8s/helm: update NOTES.txt	2019-06-03 17:53:36 +03:00
Aliaksandr Valialkin	cf75d1f0fc	README.md: mention that unused snapshots must be deleted	2019-06-03 17:26:35 +03:00
Aliaksandr Valialkin	a06b7f7f84	app/vmselect/netstorage: remove spammy error message when certain vmstorage nodes are unavailable during query execution The amount of partial responses may be tracked by `vm_partial_search_results_total` metric.	2019-06-03 17:09:50 +03:00
Aliaksandr Valialkin	1d87abc8eb	lib/procutil: typo fix in comment to WaitForSigterm	2019-06-03 16:54:37 +03:00
Aliaksandr Valialkin	a2986cde70	lib/storage: tune updating a map with today`s metric ids - Increase update iterval from 1s to 10s. This should reduce CPU usage for large amounts of metric ids with constant churn. - Reduce pendingTodayMetricIDsLock lock duration during the update.	2019-06-02 22:00:13 +03:00
Aliaksandr Valialkin	e27fd5148a	lib/storage: speed up checking metricID existence in the list for the current date	2019-06-02 18:34:20 +03:00
Aliaksandr Valialkin	d7bafde77e	vendor: update deps with `make vendor-update`	2019-06-01 23:40:56 +03:00
Aliaksandr Valialkin	53242105fb	app/vmselect/promql: allow escaping identifiers with `\` and `\xXX` Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/42	2019-05-31 17:35:54 +03:00
Aliaksandr Valialkin	25269682c2	deployment: update docker images	2019-05-29 17:44:14 +03:00
Aliaksandr Valialkin	950310d1c3	Add sections about replication and backups	2019-05-29 13:14:01 +03:00
Aliaksandr Valialkin	ee776ca8fc	app/vminsert: add `-maxConcurrentInserts` command-line flag for limiting the number of concurrent inserts	2019-05-29 12:40:22 +03:00
Aliaksandr Valialkin	a1289d7343	Makefile: run `go vet` with `-mod=vendor` in order to disable downloading vendored deps	2019-05-29 01:38:24 +03:00
Aliaksandr Valialkin	a4ec139a4a	app/vminsert: reduce memory usage for Influx, Graphite and OpenTSDB protocols Do not buffer per-connection data and just store it as it arrives	2019-05-28 18:47:52 +03:00
Aliaksandr Valialkin	a6d02ff275	lib/timerpool: use timer pool in concurrency limiters This should reduce the number of memory allocations in highly loaded system	2019-05-28 17:30:10 +03:00
Aliaksandr Valialkin	6e90aaeb8c	Makefile: add `-mod=vendor` to `go test`, so tests use external deps from `vendor` folder	2019-05-27 00:35:59 +03:00
Aliaksandr Valialkin	3b52adaf3f	Makefile: pass `GO111MODULE=on` to all the `go` invocations	2019-05-26 23:23:21 +03:00
Aliaksandr Valialkin	c944de68cd	vendor: update dependencies with `make vendor-update`	2019-05-26 23:18:42 +03:00
Aliaksandr Valialkin	b7a91d6ba7	app/vmselect: update comment according to the updated code	2019-05-26 22:39:09 +03:00
Aliaksandr Valialkin	15d1e15ae6	app/vminsert/influx: try converting string values to numeric values, since Influx agents may send numeric values as strings Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/34	2019-05-26 22:12:55 +03:00
Aliaksandr Valialkin	a2c71f18a3	app/vmselect/promql: misspeling fix	2019-05-25 21:53:48 +03:00
Aliaksandr Valialkin	bdf696ef18	all: fix misspellings	2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin	121a920a18	Makefile: add -s flag to `go fmt` in `make fmt` command	2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin	a10d27eccd	README.md: mention that new `vmstorage` node must have the same `-retentionPeriod` as the previous nodes	2019-05-25 17:18:44 +03:00
Aliaksandr Valialkin	c254adba7c	README.md: mention that VictoriaMetrics is scalable	2019-05-25 17:09:15 +03:00
Aliaksandr Valialkin	affeb677cc	README.md: mention that the majority of users should use single-node version instead of cluster version	2019-05-25 14:09:17 +03:00
Aliaksandr Valialkin	2ff996e276	app/vmselect: log slow queries if their execution time exceeds `-search.logSlowQueryDuration`	2019-05-24 16:14:46 +03:00
Aliaksandr Valialkin	628708ad76	app/vmselect: consume resultsCh data in exportHandler if writeResponseFunc failed to consume it	2019-05-24 14:54:54 +03:00
Aliaksandr Valialkin	209ad975ae	deployment: update docker images	2019-05-24 13:16:20 +03:00
Aliaksandr Valialkin	9b64dfee4b	lib/encoding: add `vm_zstd_block_{compress\|decompress}_calls_total` for determining the number CompressZSTD / DecompressZSTD calls	2019-05-24 13:01:15 +03:00
Aliaksandr Valialkin	364f4ec3bb	all: remove `-p XXXX:XXXX` from `docker run` options, since it is unnesessary if `--net=host` is set	2019-05-24 12:53:12 +03:00
Aliaksandr Valialkin	f37903adb3	app/vminsert: add `-rpc.disableCompression` command-line flag for reducing CPU usage at the cost of higher network bandwidth usage	2019-05-24 12:51:07 +03:00
Aliaksandr Valialkin	b23352dc9e	lib/encoding: add `vm_zstd_block_{original\|compressed}_bytes_total` metrics for rough estimation of block compression ratio	2019-05-24 12:34:51 +03:00
Aliaksandr Valialkin	f67f40d63a	lib/encoding: substitute CompressZSTD with CompressZSTDLevel	2019-05-24 12:32:49 +03:00
Aliaksandr Valialkin	a26e774eca	lib/httpserver: add `-http.disableResponseCompression` flag, which may help saving CPU resources at the cost of higher network bandwidth usage	2019-05-24 12:20:36 +03:00
Aliaksandr Valialkin	8e3eb5b39d	app/vmselect/promql: add `alias(q, name)` function that sets the given name to all the time series in q	2019-05-24 02:42:10 +03:00
Aliaksandr Valialkin	820cdae88d	lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29	2019-05-23 17:18:50 +03:00
Aliaksandr Valialkin	bb048937bc	app/vmselect/promql: add `label_transform(q, label, regexp, replacement)` function for replacing all the occurences of regexp with replacement in the given label for q	2019-05-23 16:26:07 +03:00
Aliaksandr Valialkin	54346de548	README.md: typo fix	2019-05-23 02:25:54 +03:00
Aliaksandr Valialkin	b98789ae9f	README.md: mention that VictoriaMetrics is high-perf cost-effective TSDB	2019-05-23 00:41:08 +03:00
Aliaksandr Valialkin	24578b4bb1	all: open-sourcing cluster version	2019-05-23 00:25:38 +03:00

1907 changed files with 44834 additions and 125770 deletions

2

.gitignore vendored

View File

@@ -7,7 +7,6 @@
 .vscode
 *.test
 *.swp
 /vmdocs
 /gocache-for-docker
 /victoria-logs-data
 /victoria-metrics-data
@@ -26,4 +25,3 @@ _site
 /docs/.jekyll-metadata
 coverage.txt
 cspell.json
 *~

									
										1

.wwhrd.yml
									
												View File
												
				@@ -4,4 +4,3 @@ allowlist:

				  - BSD-3-Clause

				  - BSD-2-Clause

				  - ISC

				  - MPL-2.0

									
										601

Makefile
									
												View File
												
				@@ -5,7 +5,7 @@ MAKE_PARALLEL := $(MAKE) -j $(MAKE_CONCURRENCY)

				DATEINFO_TAG ?= $(shell date -u +'%Y%m%d-%H%M%S')

				BUILDINFO_TAG ?= $(shell echo $$(git describe --long --all | tr '/' '-')$$( \

					      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -d' ' -f2 | cut -c 1-8)))

				LATEST_TAG ?= latest

				LATEST_TAG ?= cluster-latest

				PKG_TAG ?= $(shell git tag -l --points-at HEAD)

				ifeq ($(PKG_TAG),)

				@@ -13,7 +13,6 @@ PKG_TAG := $(BUILDINFO_TAG)

				endif

				GO_BUILDINFO = -X '$(PKG_PREFIX)/lib/buildinfo.Version=$(APP_NAME)-$(DATEINFO_TAG)-$(BUILDINFO_TAG)'

				TAR_OWNERSHIP ?= --owner=1000 --group=1000

				.PHONY: $(MAKECMDGOALS)

				@@ -25,176 +24,78 @@ include dashboards/Makefile

				include package/release/Makefile

				all: \

					victoria-metrics-prod \

					victoria-logs-prod \

					vlogscli-prod \

					vmagent-prod \

					vmalert-prod \

					vmalert-tool-prod \

					vmauth-prod \

					vmbackup-prod \

					vmrestore-prod \

					vmctl-prod

					vminsert \

					vmselect \

					vmstorage

				all-pure: \

					vminsert-pure \

					vmselect-pure \

					vmstorage-pure

				clean:

					rm -rf bin/*

				vmcluster-linux-amd64: \

					vminsert-linux-amd64 \

					vmselect-linux-amd64 \

					vmstorage-linux-amd64

				vmcluster-linux-arm64: \

					vminsert-linux-arm64 \

					vmselect-linux-arm64 \

					vmstorage-linux-arm64

				vmcluster-linux-arm: \

					vminsert-linux-arm \

					vmselect-linux-arm \

					vmstorage-linux-arm

				vmcluster-linux-ppc64le: \

					vminsert-linux-ppc64le \

					vmselect-linux-ppc64le \

					vmstorage-linux-ppc64le

				vmcluster-linux-386: \

					vminsert-linux-386 \

					vmselect-linux-386 \

					vmstorage-linux-386

				vmcluster-freebsd-amd64: \

					vminsert-freebsd-amd64 \

					vmselect-freebsd-amd64 \

					vmstorage-freebsd-amd64

				vmcluster-openbsd-amd64: \

					vminsert-openbsd-amd64 \

					vmselect-openbsd-amd64 \

					vmstorage-openbsd-amd64

				vmcluster-windows-amd64: \

					vminsert-windows-amd64 \

					vmselect-windows-amd64 \

					vmstorage-windows-amd64

				crossbuild: vmcluster-crossbuild

				vmcluster-crossbuild:

					$(MAKE_PARALLEL) vmcluster-linux-amd64 \

						vmcluster-linux-arm64 \

						vmcluster-linux-arm \

						vmcluster-linux-ppc64le \

						vmcluster-linux-386 \

						vmcluster-freebsd-amd64 \

						vmcluster-openbsd-amd64

				publish: \

					publish-victoria-metrics \

					publish-vmagent \

					publish-vmalert \

					publish-vmalert-tool \

					publish-vmauth \

					publish-vmbackup \

					publish-vmrestore \

					publish-vmctl

					publish-vminsert \

					publish-vmselect \

					publish-vmstorage

				package: \

					package-victoria-metrics \

					package-victoria-logs \

					package-vlogscli \

					package-vmagent \

					package-vmalert \

					package-vmalert-tool \

					package-vmauth \

					package-vmbackup \

					package-vmrestore \

					package-vmctl

				vmutils: \

					vmagent \

					vmalert \

					vmalert-tool \

					vmauth \

					vmbackup \

					vmrestore \

					vmctl

				vmutils-pure: \

					vmagent-pure \

					vmalert-pure \

					vmalert-tool-pure \

					vmauth-pure \

					vmbackup-pure \

					vmrestore-pure \

					vmctl-pure

				vmutils-linux-amd64: \

					vmagent-linux-amd64 \

					vmalert-linux-amd64 \

					vmalert-tool-linux-amd64 \

					vmauth-linux-amd64 \

					vmbackup-linux-amd64 \

					vmrestore-linux-amd64 \

					vmctl-linux-amd64

				vmutils-linux-arm64: \

					vmagent-linux-arm64 \

					vmalert-linux-arm64 \

					vmalert-tool-linux-arm64 \

					vmauth-linux-arm64 \

					vmbackup-linux-arm64 \

					vmrestore-linux-arm64 \

					vmctl-linux-arm64

				vmutils-linux-arm: \

					vmagent-linux-arm \

					vmalert-linux-arm \

					vmalert-tool-linux-arm \

					vmauth-linux-arm \

					vmbackup-linux-arm \

					vmrestore-linux-arm \

					vmctl-linux-arm

				vmutils-linux-386: \

					vmagent-linux-386 \

					vmalert-linux-386 \

					vmalert-tool-linux-386 \

					vmauth-linux-386 \

					vmbackup-linux-386 \

					vmrestore-linux-386 \

					vmctl-linux-386

				vmutils-linux-ppc64le: \

					vmagent-linux-ppc64le \

					vmalert-linux-ppc64le \

					vmalert-tool-linux-ppc64le \

					vmauth-linux-ppc64le \

					vmbackup-linux-ppc64le \

					vmrestore-linux-ppc64le \

					vmctl-linux-ppc64le

				vmutils-darwin-amd64: \

					vmagent-darwin-amd64 \

					vmalert-darwin-amd64 \

					vmalert-tool-darwin-amd64 \

					vmauth-darwin-amd64 \

					vmbackup-darwin-amd64 \

					vmrestore-darwin-amd64 \

					vmctl-darwin-amd64

				vmutils-darwin-arm64: \

					vmagent-darwin-arm64 \

					vmalert-darwin-arm64 \

					vmalert-tool-darwin-arm64 \

					vmauth-darwin-arm64 \

					vmbackup-darwin-arm64 \

					vmrestore-darwin-arm64 \

					vmctl-darwin-arm64

				vmutils-freebsd-amd64: \

					vmagent-freebsd-amd64 \

					vmalert-freebsd-amd64 \

					vmalert-tool-freebsd-amd64 \

					vmauth-freebsd-amd64 \

					vmbackup-freebsd-amd64 \

					vmrestore-freebsd-amd64 \

					vmctl-freebsd-amd64

				vmutils-openbsd-amd64: \

					vmagent-openbsd-amd64 \

					vmalert-openbsd-amd64 \

					vmalert-tool-openbsd-amd64 \

					vmauth-openbsd-amd64 \

					vmbackup-openbsd-amd64 \

					vmrestore-openbsd-amd64 \

					vmctl-openbsd-amd64

				vmutils-windows-amd64: \

					vmagent-windows-amd64 \

					vmalert-windows-amd64 \

					vmalert-tool-windows-amd64 \

					vmauth-windows-amd64 \

					vmbackup-windows-amd64 \

					vmrestore-windows-amd64 \

					vmctl-windows-amd64

				crossbuild:

					$(MAKE_PARALLEL) victoria-metrics-crossbuild vmutils-crossbuild

				victoria-metrics-crossbuild: \

					victoria-metrics-linux-386 \

					victoria-metrics-linux-amd64 \

					victoria-metrics-linux-arm64 \

					victoria-metrics-linux-arm \

					victoria-metrics-linux-ppc64le \

					victoria-metrics-darwin-amd64 \

					victoria-metrics-darwin-arm64 \

					victoria-metrics-freebsd-amd64 \

					victoria-metrics-openbsd-amd64 \

					victoria-metrics-windows-amd64

				vmutils-crossbuild: \

					vmutils-linux-386 \

					vmutils-linux-amd64 \

					vmutils-linux-arm64 \

					vmutils-linux-arm \

					vmutils-linux-ppc64le \

					vmutils-darwin-amd64 \

					vmutils-darwin-arm64 \

					vmutils-freebsd-amd64 \

					vmutils-openbsd-amd64 \

					vmutils-windows-amd64

					package-vminsert \

					package-vmselect \

					package-vmstorage

				publish-release:

					rm -rf bin/*

				@@ -204,296 +105,67 @@ publish-release:

						git checkout $(TAG)-enterprise-cluster && $(MAKE) release && LATEST_TAG=enterprise-cluster-stable $(MAKE) publish

				release:

					$(MAKE_PARALLEL) \

						release-victoria-metrics \

						release-vmutils

					$(MAKE_PARALLEL) release-vmcluster

				release-victoria-metrics: \

					release-victoria-metrics-linux-386 \

					release-victoria-metrics-linux-amd64 \

					release-victoria-metrics-linux-arm \

					release-victoria-metrics-linux-arm64 \

					release-victoria-metrics-darwin-amd64 \

					release-victoria-metrics-darwin-arm64 \

					release-victoria-metrics-freebsd-amd64 \

					release-victoria-metrics-openbsd-amd64 \

					release-victoria-metrics-windows-amd64

				release-vmcluster: \

					release-vmcluster-linux-amd64 \

					release-vmcluster-linux-arm64 \

					release-vmcluster-freebsd-amd64 \

					release-vmcluster-openbsd-amd64 \

					release-vmcluster-windows-amd64

				release-victoria-metrics-linux-386:

					GOOS=linux GOARCH=386 $(MAKE) release-victoria-metrics-goos-goarch

				release-vmcluster-linux-amd64:

					GOOS=linux GOARCH=amd64 $(MAKE) release-vmcluster-goos-goarch

				release-victoria-metrics-linux-amd64:

					GOOS=linux GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch

				release-vmcluster-linux-arm64:

					GOOS=linux GOARCH=arm64 $(MAKE) release-vmcluster-goos-goarch

				release-victoria-metrics-linux-arm:

					GOOS=linux GOARCH=arm $(MAKE) release-victoria-metrics-goos-goarch

				release-vmcluster-freebsd-amd64:

					GOOS=freebsd GOARCH=amd64 $(MAKE) release-vmcluster-goos-goarch

				release-victoria-metrics-linux-arm64:

					GOOS=linux GOARCH=arm64 $(MAKE) release-victoria-metrics-goos-goarch

				release-vmcluster-openbsd-amd64:

					GOOS=openbsd GOARCH=amd64 $(MAKE) release-vmcluster-goos-goarch

				release-victoria-metrics-darwin-amd64:

					GOOS=darwin GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch

				release-vmcluster-windows-amd64:

					GOARCH=amd64 $(MAKE) release-vmcluster-windows-goarch

				release-victoria-metrics-darwin-arm64:

					GOOS=darwin GOARCH=arm64 $(MAKE) release-victoria-metrics-goos-goarch

				release-victoria-metrics-freebsd-amd64:

					GOOS=freebsd GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch

				release-victoria-metrics-openbsd-amd64:

					GOOS=openbsd GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch

				release-victoria-metrics-windows-amd64:

					GOARCH=amd64 $(MAKE) release-victoria-metrics-windows-goarch

				release-victoria-metrics-goos-goarch: victoria-metrics-$(GOOS)-$(GOARCH)-prod

				release-vmcluster-goos-goarch: \

					vminsert-$(GOOS)-$(GOARCH)-prod \

					vmselect-$(GOOS)-$(GOARCH)-prod \

					vmstorage-$(GOOS)-$(GOARCH)-prod

					cd bin && \

						tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							victoria-metrics-$(GOOS)-$(GOARCH)-prod \

						tar --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							vminsert-$(GOOS)-$(GOARCH)-prod \

							vmselect-$(GOOS)-$(GOARCH)-prod \

							vmstorage-$(GOOS)-$(GOARCH)-prod \

						&& sha256sum victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							victoria-metrics-$(GOOS)-$(GOARCH)-prod \

							vminsert-$(GOOS)-$(GOARCH)-prod \

							vmselect-$(GOOS)-$(GOARCH)-prod \

							vmstorage-$(GOOS)-$(GOARCH)-prod \

							| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf victoria-metrics-$(GOOS)-$(GOARCH)-prod

					cd bin && rm -rf \

						vminsert-$(GOOS)-$(GOARCH)-prod \

						vmselect-$(GOOS)-$(GOARCH)-prod \

						vmstorage-$(GOOS)-$(GOARCH)-prod

				release-victoria-metrics-windows-goarch: victoria-metrics-windows-$(GOARCH)-prod

				release-vmcluster-windows-goarch: \

					vminsert-windows-$(GOARCH)-prod \

					vmselect-windows-$(GOARCH)-prod \

					vmstorage-windows-$(GOARCH)-prod

					cd bin && \

						zip victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \

							victoria-metrics-windows-$(GOARCH)-prod.exe \

							vminsert-windows-$(GOARCH)-prod.exe \

							vmselect-windows-$(GOARCH)-prod.exe \

							vmstorage-windows-$(GOARCH)-prod.exe \

						&& sha256sum victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \

							victoria-metrics-windows-$(GOARCH)-prod.exe \

							> victoria-metrics-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt

							vminsert-windows-$(GOARCH)-prod.exe \

							vmselect-windows-$(GOARCH)-prod.exe \

							vmstorage-windows-$(GOARCH)-prod.exe \

						> victoria-metrics-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf \

						victoria-metrics-windows-$(GOARCH)-prod.exe

				release-victoria-logs-bundle: \

					release-victoria-logs \

					release-vlogscli

				publish-victoria-logs-bundle: \

					publish-victoria-logs \

					publish-vlogscli

				release-victoria-logs:

					$(MAKE_PARALLEL) release-victoria-logs-linux-386 \

						release-victoria-logs-linux-amd64 \

						release-victoria-logs-linux-arm \

						release-victoria-logs-linux-arm64 \

						release-victoria-logs-darwin-amd64 \

						release-victoria-logs-darwin-arm64 \

						release-victoria-logs-freebsd-amd64 \

						release-victoria-logs-openbsd-amd64 \

						release-victoria-logs-windows-amd64

				release-victoria-logs-linux-386:

					GOOS=linux GOARCH=386 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-linux-amd64:

					GOOS=linux GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-linux-arm:

					GOOS=linux GOARCH=arm $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-linux-arm64:

					GOOS=linux GOARCH=arm64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-darwin-amd64:

					GOOS=darwin GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-darwin-arm64:

					GOOS=darwin GOARCH=arm64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-freebsd-amd64:

					GOOS=freebsd GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-openbsd-amd64:

					GOOS=openbsd GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch

				release-victoria-logs-windows-amd64:

					GOARCH=amd64 $(MAKE) release-victoria-logs-windows-goarch

				release-victoria-logs-goos-goarch: victoria-logs-$(GOOS)-$(GOARCH)-prod

					cd bin && \

						tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							victoria-logs-$(GOOS)-$(GOARCH)-prod \

						&& sha256sum victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							victoria-logs-$(GOOS)-$(GOARCH)-prod \

							| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf victoria-logs-$(GOOS)-$(GOARCH)-prod

				release-victoria-logs-windows-goarch: victoria-logs-windows-$(GOARCH)-prod

					cd bin && \

						zip victoria-logs-windows-$(GOARCH)-$(PKG_TAG).zip \

							victoria-logs-windows-$(GOARCH)-prod.exe \

						&& sha256sum victoria-logs-windows-$(GOARCH)-$(PKG_TAG).zip \

							victoria-logs-windows-$(GOARCH)-prod.exe \

							> victoria-logs-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf \

						victoria-logs-windows-$(GOARCH)-prod.exe

				release-vlogscli:

					$(MAKE_PARALLEL) release-vlogscli-linux-386 \

						release-vlogscli-linux-amd64 \

						release-vlogscli-linux-arm \

						release-vlogscli-linux-arm64 \

						release-vlogscli-darwin-amd64 \

						release-vlogscli-darwin-arm64 \

						release-vlogscli-freebsd-amd64 \

						release-vlogscli-openbsd-amd64 \

						release-vlogscli-windows-amd64

				release-vlogscli-linux-386:

					GOOS=linux GOARCH=386 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-linux-amd64:

					GOOS=linux GOARCH=amd64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-linux-arm:

					GOOS=linux GOARCH=arm $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-linux-arm64:

					GOOS=linux GOARCH=arm64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-darwin-amd64:

					GOOS=darwin GOARCH=amd64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-darwin-arm64:

					GOOS=darwin GOARCH=arm64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-freebsd-amd64:

					GOOS=freebsd GOARCH=amd64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-openbsd-amd64:

					GOOS=openbsd GOARCH=amd64 $(MAKE) release-vlogscli-goos-goarch

				release-vlogscli-windows-amd64:

					GOARCH=amd64 $(MAKE) release-vlogscli-windows-goarch

				release-vlogscli-goos-goarch: vlogscli-$(GOOS)-$(GOARCH)-prod

					cd bin && \

						tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vlogscli-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							vlogscli-$(GOOS)-$(GOARCH)-prod \

						&& sha256sum vlogscli-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							vlogscli-$(GOOS)-$(GOARCH)-prod \

							| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > vlogscli-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf vlogscli-$(GOOS)-$(GOARCH)-prod

				release-vlogscli-windows-goarch: vlogscli-windows-$(GOARCH)-prod

					cd bin && \

						zip vlogscli-windows-$(GOARCH)-$(PKG_TAG).zip \

							vlogscli-windows-$(GOARCH)-prod.exe \

						&& sha256sum vlogscli-windows-$(GOARCH)-$(PKG_TAG).zip \

							vlogscli-windows-$(GOARCH)-prod.exe \

							> vlogscli-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf \

						vlogscli-windows-$(GOARCH)-prod.exe

				release-vmutils: \

					release-vmutils-linux-386 \

					release-vmutils-linux-amd64 \

					release-vmutils-linux-arm64 \

					release-vmutils-linux-arm \

					release-vmutils-darwin-amd64 \

					release-vmutils-darwin-arm64 \

					release-vmutils-freebsd-amd64 \

					release-vmutils-openbsd-amd64 \

					release-vmutils-windows-amd64

				release-vmutils-linux-386:

					GOOS=linux GOARCH=386 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-linux-amd64:

					GOOS=linux GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-linux-arm64:

					GOOS=linux GOARCH=arm64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-linux-arm:

					GOOS=linux GOARCH=arm $(MAKE) release-vmutils-goos-goarch

				release-vmutils-darwin-amd64:

					GOOS=darwin GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-darwin-arm64:

					GOOS=darwin GOARCH=arm64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-freebsd-amd64:

					GOOS=freebsd GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-openbsd-amd64:

					GOOS=openbsd GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch

				release-vmutils-windows-amd64:

					GOARCH=amd64 $(MAKE) release-vmutils-windows-goarch

				release-vmutils-goos-goarch: \

					vmagent-$(GOOS)-$(GOARCH)-prod \

					vmalert-$(GOOS)-$(GOARCH)-prod \

					vmalert-tool-$(GOOS)-$(GOARCH)-prod \

					vmauth-$(GOOS)-$(GOARCH)-prod \

					vmbackup-$(GOOS)-$(GOARCH)-prod \

					vmrestore-$(GOOS)-$(GOARCH)-prod \

					vmctl-$(GOOS)-$(GOARCH)-prod

					cd bin && \

						tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							vmagent-$(GOOS)-$(GOARCH)-prod \

							vmalert-$(GOOS)-$(GOARCH)-prod \

							vmalert-tool-$(GOOS)-$(GOARCH)-prod \

							vmauth-$(GOOS)-$(GOARCH)-prod \

							vmbackup-$(GOOS)-$(GOARCH)-prod \

							vmrestore-$(GOOS)-$(GOARCH)-prod \

							vmctl-$(GOOS)-$(GOARCH)-prod \

						&& sha256sum vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \

							vmagent-$(GOOS)-$(GOARCH)-prod \

							vmalert-$(GOOS)-$(GOARCH)-prod \

							vmalert-tool-$(GOOS)-$(GOARCH)-prod \

							vmauth-$(GOOS)-$(GOARCH)-prod \

							vmbackup-$(GOOS)-$(GOARCH)-prod \

							vmrestore-$(GOOS)-$(GOARCH)-prod \

							vmctl-$(GOOS)-$(GOARCH)-prod \

							| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf \

						vmagent-$(GOOS)-$(GOARCH)-prod \

						vmalert-$(GOOS)-$(GOARCH)-prod \

						vmalert-tool-$(GOOS)-$(GOARCH)-prod \

						vmauth-$(GOOS)-$(GOARCH)-prod \

						vmbackup-$(GOOS)-$(GOARCH)-prod \

						vmrestore-$(GOOS)-$(GOARCH)-prod \

						vmctl-$(GOOS)-$(GOARCH)-prod

				release-vmutils-windows-goarch: \

					vmagent-windows-$(GOARCH)-prod \

					vmalert-windows-$(GOARCH)-prod \

					vmalert-tool-windows-$(GOARCH)-prod \

					vmauth-windows-$(GOARCH)-prod \

					vmbackup-windows-$(GOARCH)-prod \

					vmrestore-windows-$(GOARCH)-prod \

					vmctl-windows-$(GOARCH)-prod

					cd bin && \

						zip vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \

							vmagent-windows-$(GOARCH)-prod.exe \

							vmalert-windows-$(GOARCH)-prod.exe \

							vmalert-tool-windows-$(GOARCH)-prod.exe \

							vmauth-windows-$(GOARCH)-prod.exe \

							vmbackup-windows-$(GOARCH)-prod.exe \

							vmrestore-windows-$(GOARCH)-prod.exe \

							vmctl-windows-$(GOARCH)-prod.exe \

						&& sha256sum vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \

							vmagent-windows-$(GOARCH)-prod.exe \

							vmalert-windows-$(GOARCH)-prod.exe \

							vmalert-tool-windows-$(GOARCH)-prod.exe \

							vmauth-windows-$(GOARCH)-prod.exe \

							vmbackup-windows-$(GOARCH)-prod.exe \

							vmrestore-windows-$(GOARCH)-prod.exe \

							vmctl-windows-$(GOARCH)-prod.exe \

							> vmutils-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt

					cd bin && rm -rf \

						vmagent-windows-$(GOARCH)-prod.exe \

						vmalert-windows-$(GOARCH)-prod.exe \

						vmalert-tool-windows-$(GOARCH)-prod.exe \

						vmauth-windows-$(GOARCH)-prod.exe \

						vmbackup-windows-$(GOARCH)-prod.exe \

						vmrestore-windows-$(GOARCH)-prod.exe \

						vmctl-windows-$(GOARCH)-prod.exe

						vminsert-windows-$(GOARCH)-prod.exe \

						vmselect-windows-$(GOARCH)-prod.exe \

						vmstorage-windows-$(GOARCH)-prod.exe

				pprof-cpu:

					go tool pprof -trim_path=github.com/VictoriaMetrics/VictoriaMetrics@ $(PPROF_FILE)

				@@ -511,19 +183,19 @@ check-all: fmt vet golangci-lint govulncheck

				clean-checkers: remove-golangci-lint remove-govulncheck

				test:

					DISABLE_FSYNC_FOR_TESTING=1 go test ./lib/... ./app/...

					go test ./lib/... ./app/...

				test-race:

					DISABLE_FSYNC_FOR_TESTING=1 go test -race ./lib/... ./app/...

					go test -race ./lib/... ./app/...

				test-pure:

					DISABLE_FSYNC_FOR_TESTING=1 CGO_ENABLED=0 go test ./lib/... ./app/...

					CGO_ENABLED=0 go test ./lib/... ./app/...

				test-full:

					DISABLE_FSYNC_FOR_TESTING=1 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...

					go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...

				test-full-386:

					DISABLE_FSYNC_FOR_TESTING=1 GOARCH=386 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...

					GOARCH=386 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...

				benchmark:

					go test -bench=. ./lib/...

				@@ -534,9 +206,9 @@ benchmark-pure:

					CGO_ENABLED=0 go test -bench=. ./app/...

				vendor-update:

					go get -u ./lib/...

					go get -u ./app/...

					go mod tidy -compat=1.23

					go get -u -d ./lib/...

					go get -u -d ./app/...

					go mod tidy -compat=1.22

					go mod vendor

				app-local:

				@@ -562,7 +234,7 @@ golangci-lint: install-golangci-lint

					golangci-lint run

				install-golangci-lint:

					which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.60.3

					which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.59.1

				remove-golangci-lint:

					rm -rf `which golangci-lint`

				@@ -581,3 +253,34 @@ install-wwhrd:

				check-licenses: install-wwhrd

					wwhrd check -f .wwhrd.yml

				copy-docs:

				# The 'printf' function is used instead of 'echo' or 'echo -e' to handle line breaks (e.g. '\n') in the same way on different operating systems (MacOS/Ubuntu Linux/Arch Linux) and their shells (bash/sh/zsh/fish).

				# For details, see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4548#issue-1782796419 and https://stackoverflow.com/questions/8467424/echo-newline-in-bash-prints-literal-n

					echo "---" > ${DST}

					@if [ ${ORDER} -ne 0 ]; then \

						echo "sort: ${ORDER}" >> ${DST}; \

						echo "weight: ${ORDER}" >> ${DST}; \

						printf "menu:\n  docs:\n    parent: 'victoriametrics'\n    weight: ${ORDER}\n" >> ${DST}; \

					fi

					echo "title: ${TITLE}" >> ${DST}

					@if [ ${OLD_URL} ]; then \

						printf "aliases:\n  - ${OLD_URL}\n" >> ${DST}; \

					fi

					echo "---" >> ${DST}

					cat ${SRC} >> ${DST}

					sed -i='.tmp' 's/<img src=\"docs\//<img src=\"\//' ${DST}

					sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"\//' ${DST}

					sed -i='.tmp' 's/](docs\//](/' ${DST}

					rm -rf docs/*.tmp

				# Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site.

				# For ORDER=0 it adds no order tag/weight tag.

				# FOR OLD_URL - relative link, used for backward compatibility with the link from documentation based on GitHub pages (old one)

				# FOR OLD_URL='' it adds no alias, it should be empty for every new page, don't change it for already existing links.

				# Images starting with <img src="docs/ are replaced with <img src="

				# Cluster docs are supposed to be ordered as 2nd.

				# The rest of docs is ordered manually.

				docs-sync:

					SRC=README.md DST=docs/Cluster-VictoriaMetrics.md OLD_URL='/Cluster-VictoriaMetrics.html' ORDER=2 TITLE='Cluster version' $(MAKE) copy-docs

1954

README.md

View File

File diff suppressed because it is too large Load Diff

									
										4

SECURITY.md
									
												View File
												
				@@ -7,12 +7,12 @@ The following versions of VictoriaMetrics receive regular security fixes:

				| Version | Supported          |

				|---------|--------------------|

				| [latest release](https://docs.victoriametrics.com/changelog/) | :white_check_mark: |

				| v1.102.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |

				| v1.97.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |

				| v1.93.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |

				| other releases  | :x:                |

				See [this page](https://victoriametrics.com/security/) for more details.

				## Reporting a Vulnerability

				Please report any security issues to <security@victoriametrics.com>

				Please report any security issues to security@victoriametrics.com

BIN
VM_logo.zip

View File

Binary file not shown.

									
										4

app/victoria-logs/deployment/Dockerfile
									
												View File
												
				@@ -1,8 +1,8 @@

				ARG base_image=non-existing

				ARG base_image

				FROM $base_image

				EXPOSE 9428

				ENTRYPOINT ["/victoria-logs-prod"]

				ARG src_binary=non-existing

				ARG src_binary

				COPY $src_binary ./victoria-logs-prod

									
										3

app/victoria-logs/main.go
									
												View File
												
				@@ -92,9 +92,6 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {

					if vlselect.RequestHandler(w, r) {

						return true

					}

					if vlstorage.RequestHandler(w, r) {

						return true

					}

					return false

				}

									
										4

app/victoria-logs/multiarch/Dockerfile
									
												View File
												
				@@ -1,6 +1,6 @@

				# See https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b

				ARG certs_image=non-existing

				ARG root_image=non-existing

				ARG certs_image

				ARG root_image

				FROM $certs_image AS certs

				RUN apk update && apk upgrade && apk --update --no-cache add ca-certificates

									
										149

app/victoria-metrics/Makefile
									
												View File
											
				@@ -1,149 +0,0 @@

				# All these commands must run from repository root.

				victoria-metrics:

					APP_NAME=victoria-metrics $(MAKE) app-local

				victoria-metrics-race:

					APP_NAME=victoria-metrics RACE=-race $(MAKE) app-local

				victoria-metrics-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker

				victoria-metrics-pure-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-pure

				victoria-metrics-linux-amd64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-amd64

				victoria-metrics-linux-arm-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-arm

				victoria-metrics-linux-arm64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-arm64

				victoria-metrics-linux-ppc64le-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-ppc64le

				victoria-metrics-linux-386-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-386

				victoria-metrics-darwin-amd64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-darwin-amd64

				victoria-metrics-darwin-arm64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-darwin-arm64

				victoria-metrics-freebsd-amd64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-freebsd-amd64

				victoria-metrics-openbsd-amd64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-openbsd-amd64

				victoria-metrics-windows-amd64-prod:

					APP_NAME=victoria-metrics $(MAKE) app-via-docker-windows-amd64

				package-victoria-metrics:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker

				package-victoria-metrics-pure:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-pure

				package-victoria-metrics-amd64:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-amd64

				package-victoria-metrics-arm:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-arm

				package-victoria-metrics-arm64:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-arm64

				package-victoria-metrics-ppc64le:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-ppc64le

				package-victoria-metrics-386:

					APP_NAME=victoria-metrics $(MAKE) package-via-docker-386

				publish-victoria-metrics:

					APP_NAME=victoria-metrics $(MAKE) publish-via-docker

				run-victoria-metrics:

					mkdir -p victoria-metrics-data

					DOCKER_OPTS='-v $(shell pwd)/victoria-metrics-data:/victoria-metrics-data' \

					APP_NAME=victoria-metrics \

					ARGS='-graphiteListenAddr=:2003 -opentsdbListenAddr=:4242 -retentionPeriod=12 -search.maxUniqueTimeseries=1000000 -search.maxQueryDuration=10m' \

					$(MAKE) run-via-docker

				victoria-metrics-linux-amd64:

					APP_NAME=victoria-metrics CGO_ENABLED=1 GOOS=linux GOARCH=amd64 $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-arm:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=arm $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-arm64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=arm64 $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-ppc64le:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-s390x:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-loong64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch

				victoria-metrics-linux-386:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch

				victoria-metrics-darwin-amd64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 $(MAKE) app-local-goos-goarch

				victoria-metrics-darwin-arm64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 $(MAKE) app-local-goos-goarch

				victoria-metrics-freebsd-amd64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 $(MAKE) app-local-goos-goarch

				victoria-metrics-openbsd-amd64:

					APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=openbsd GOARCH=amd64 $(MAKE) app-local-goos-goarch

				victoria-metrics-windows-amd64:

					GOARCH=amd64 APP_NAME=victoria-metrics $(MAKE) app-local-windows-goarch

				victoria-metrics-pure:

					APP_NAME=victoria-metrics $(MAKE) app-local-pure

				### Packaging as DEB - amd64

				victoria-metrics-package-deb-amd64: victoria-metrics-linux-amd64-prod

					./package/package_deb.sh amd64

				### Packaging as DEB - arm64

				victoria-metrics-package-deb-arm: victoria-metrics-linux-arm-prod

					./package/package_deb.sh arm

				### Packaging as DEB - arm64

				victoria-metrics-package-deb-arm64: victoria-metrics-linux-arm64-prod

					./package/package_deb.sh arm64

				### Packaging as DEB - all

				victoria-metrics-package-deb: \

				        victoria-metrics-package-deb-amd64 \

				        victoria-metrics-package-deb-arm \

				        victoria-metrics-package-deb-arm64

				### Packaging as RPM - amd64

				victoria-metrics-package-rpm-amd64: victoria-metrics-linux-amd64-prod

					./package/package_rpm.sh amd64

				### Packaging as RPM - arm64

				victoria-metrics-package-rpm-arm64: victoria-metrics-linux-arm64-prod

					./package/package_rpm.sh arm64

				### Packaging as RPM - all

				victoria-metrics-package-rpm: \

				        victoria-metrics-package-rpm-amd64 \

				        victoria-metrics-package-rpm-arm64

				### Packaging as both DEB and RPM - all

				victoria-metrics-package-deb-rpm: \

				        victoria-metrics-package-deb \

				        victoria-metrics-package-rpm

									
										8

app/victoria-metrics/deployment/Dockerfile
									
												View File
											
				@@ -1,8 +0,0 @@

				ARG base_image=non-existing

				FROM $base_image

				EXPOSE 8428

				ENTRYPOINT ["/victoria-metrics-prod"]

				ARG src_binary=non-existing

				COPY $src_binary ./victoria-metrics-prod

									
										154

app/victoria-metrics/main.go
									
												View File
											
				@@ -1,154 +0,0 @@

				package main

				import (

					"flag"

					"fmt"

					"net/http"

					"os"

					"time"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"

					vminsertcommon "github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/common"

					vminsertrelabel "github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"

				)

				var (

					httpListenAddrs  = flagutil.NewArrayString("httpListenAddr", "TCP addresses to listen for incoming http requests. See also -tls and -httpListenAddr.useProxyProtocol")

					useProxyProtocol = flagutil.NewArrayBool("httpListenAddr.useProxyProtocol", "Whether to use proxy protocol for connections accepted at the corresponding -httpListenAddr . "+

						"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+

						"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")

					minScrapeInterval = flag.Duration("dedup.minScrapeInterval", 0, "Leave only the last sample in every time series per each discrete interval "+

						"equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication")

					dryRun = flag.Bool("dryRun", false, "Whether to check config files without running VictoriaMetrics. The following config files are checked: "+

						"-promscrape.config, -relabelConfig and -streamAggr.config. Unknown config entries aren't allowed in -promscrape.config by default. "+

						"This can be changed with -promscrape.config.strictParse=false command-line flag")

					inmemoryDataFlushInterval = flag.Duration("inmemoryDataFlushInterval", 5*time.Second, "The interval for guaranteed saving of in-memory data to disk. "+

						"The saved data survives unclean shutdowns such as OOM crash, hardware reset, SIGKILL, etc. "+

						"Bigger intervals may help increase the lifetime of flash storage with limited write cycles (e.g. Raspberry PI). "+

						"Smaller intervals increase disk IO load. Minimum supported value is 1s")

				)

				func main() {

					// Write flags and help message to stdout, since it is easier to grep or pipe.

					flag.CommandLine.SetOutput(os.Stdout)

					flag.Usage = usage

					envflag.Parse()

					buildinfo.Init()

					logger.Init()

					if promscrape.IsDryRun() {

						*dryRun = true

					}

					if *dryRun {

						if err := promscrape.CheckConfig(); err != nil {

							logger.Fatalf("error when checking -promscrape.config: %s", err)

						}

						if err := vminsertrelabel.CheckRelabelConfig(); err != nil {

							logger.Fatalf("error when checking -relabelConfig: %s", err)

						}

						if err := vminsertcommon.CheckStreamAggrConfig(); err != nil {

							logger.Fatalf("error when checking -streamAggr.config: %s", err)

						}

						logger.Infof("-promscrape.config is ok; exiting with 0 status code")

						return

					}

					listenAddrs := *httpListenAddrs

					if len(listenAddrs) == 0 {

						listenAddrs = []string{":8428"}

					}

					logger.Infof("starting VictoriaMetrics at %q...", listenAddrs)

					startTime := time.Now()

					storage.SetDedupInterval(*minScrapeInterval)

					storage.SetDataFlushInterval(*inmemoryDataFlushInterval)

					vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)

					vmselect.Init()

					vminsert.Init()

					startSelfScraper()

					go httpserver.Serve(listenAddrs, useProxyProtocol, requestHandler)

					logger.Infof("started VictoriaMetrics in %.3f seconds", time.Since(startTime).Seconds())

					pushmetrics.Init()

					sig := procutil.WaitForSigterm()

					logger.Infof("received signal %s", sig)

					pushmetrics.Stop()

					stopSelfScraper()

					logger.Infof("gracefully shutting down webservice at %q", listenAddrs)

					startTime = time.Now()

					if err := httpserver.Stop(listenAddrs); err != nil {

						logger.Fatalf("cannot stop the webservice: %s", err)

					}

					logger.Infof("successfully shut down the webservice in %.3f seconds", time.Since(startTime).Seconds())

					vminsert.Stop()

					vmstorage.Stop()

					vmselect.Stop()

					fs.MustStopDirRemover()

					logger.Infof("the VictoriaMetrics has been stopped in %.3f seconds", time.Since(startTime).Seconds())

				}

				func requestHandler(w http.ResponseWriter, r *http.Request) bool {

					if r.URL.Path == "/" {

						if r.Method != http.MethodGet {

							return false

						}

						w.Header().Add("Content-Type", "text/html; charset=utf-8")

						fmt.Fprintf(w, "<h2>Single-node VictoriaMetrics</h2></br>")

						fmt.Fprintf(w, "See docs at <a href='https://docs.victoriametrics.com/'>https://docs.victoriametrics.com/</a></br>")

						fmt.Fprintf(w, "Useful endpoints:</br>")

						httpserver.WriteAPIHelp(w, [][2]string{

							{"vmui", "Web UI"},

							{"targets", "status for discovered active targets"},

							{"service-discovery", "labels before and after relabeling for discovered targets"},

							{"metric-relabel-debug", "debug metric relabeling"},

							{"expand-with-exprs", "WITH expressions' tutorial"},

							{"api/v1/targets", "advanced information about discovered targets in JSON format"},

							{"config", "-promscrape.config contents"},

							{"metrics", "available service metrics"},

							{"flags", "command-line flags"},

							{"api/v1/status/tsdb", "tsdb status page"},

							{"api/v1/status/top_queries", "top queries"},

							{"api/v1/status/active_queries", "active queries"},

							{"-/reload", "reload configuration"},

						})

						return true

					}

					if vminsert.RequestHandler(w, r) {

						return true

					}

					if vmselect.RequestHandler(w, r) {

						return true

					}

					if vmstorage.RequestHandler(w, r) {

						return true

					}

					return false

				}

				func usage() {

					const s = `

				victoria-metrics is a time series database and monitoring solution.

				See the docs at https://docs.victoriametrics.com/

				`

					flagutil.Usage(s)

				}

									
										619

app/victoria-metrics/main_test.go
									
												View File
											
				@@ -1,619 +0,0 @@

				package main

				import (

					"bytes"

					"encoding/json"

					"errors"

					"flag"

					"fmt"

					"io"

					"log"

					"math/rand"

					"net"

					"net/http"

					"os"

					"path/filepath"

					"reflect"

					"strconv"

					"strings"

					"testing"

					"time"

					testutil "github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/test"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"

				)

				const (

					testFixturesDir            = "testdata"

					testStorageSuffix          = "vm-test-storage"

					testHTTPListenAddr         = ":7654"

					testStatsDListenAddr       = ":2003"

					testOpenTSDBListenAddr     = ":4242"

					testOpenTSDBHTTPListenAddr = ":4243"

					testLogLevel               = "INFO"

				)

				const (

					testReadHTTPPath           = "http://127.0.0.1" + testHTTPListenAddr

					testWriteHTTPPath          = "http://127.0.0.1" + testHTTPListenAddr + "/write"

					testOpenTSDBWriteHTTPPath  = "http://127.0.0.1" + testOpenTSDBHTTPListenAddr + "/api/put"

					testPromWriteHTTPPath      = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/write"

					testImportCSVWriteHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/import/csv"

					testHealthHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/health"

				)

				const (

					testStorageInitTimeout = 10 * time.Second

				)

				var (

					storagePath   string

					insertionTime = time.Now().UTC()

				)

				type test struct {

					Name                     string   `json:"name"`

					Data                     []string `json:"data"`

					InsertQuery              string   `json:"insert_query"`

					Query                    []string `json:"query"`

					ResultMetrics            []Metric `json:"result_metrics"`

					ResultSeries             Series   `json:"result_series"`

					ResultQuery              Query    `json:"result_query"`

					Issue                    string   `json:"issue"`

					ExpectedResultLinesCount int      `json:"expected_result_lines_count"`

				}

				type Metric struct {

					Metric     map[string]string `json:"metric"`

					Values     []float64         `json:"values"`

					Timestamps []int64           `json:"timestamps"`

				}

				func (r *Metric) UnmarshalJSON(b []byte) error {

					type plain Metric

					return json.Unmarshal(testutil.PopulateTimeTpl(b, insertionTime), (*plain)(r))

				}

				type Series struct {

					Status string              `json:"status"`

					Data   []map[string]string `json:"data"`

				}

				type Query struct {

					Status string `json:"status"`

					Data   struct {

						ResultType string          `json:"resultType"`

						Result     json.RawMessage `json:"result"`

					} `json:"data"`

				}

				const rtVector, rtMatrix = "vector", "matrix"

				func (q *Query) metrics() ([]Metric, error) {

					switch q.Data.ResultType {

					case rtVector:

						var r QueryInstant

						if err := json.Unmarshal(q.Data.Result, &r.Result); err != nil {

							return nil, err

						}

						return r.metrics()

					case rtMatrix:

						var r QueryRange

						if err := json.Unmarshal(q.Data.Result, &r.Result); err != nil {

							return nil, err

						}

						return r.metrics()

					default:

						return nil, fmt.Errorf("unknown result type %q", q.Data.ResultType)

					}

				}

				type QueryInstant struct {

					Result []struct {

						Labels map[string]string `json:"metric"`

						TV     [2]any            `json:"value"`

					} `json:"result"`

				}

				func (q QueryInstant) metrics() ([]Metric, error) {

					result := make([]Metric, len(q.Result))

					for i, res := range q.Result {

						f, err := strconv.ParseFloat(res.TV[1].(string), 64)

						if err != nil {

							return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, res.TV[1], err)

						}

						var m Metric

						m.Metric = res.Labels

						m.Timestamps = append(m.Timestamps, int64(res.TV[0].(float64)))

						m.Values = append(m.Values, f)

						result[i] = m

					}

					return result, nil

				}

				type QueryRange struct {

					Result []struct {

						Metric map[string]string `json:"metric"`

						Values [][]any           `json:"values"`

					} `json:"result"`

				}

				func (q QueryRange) metrics() ([]Metric, error) {

					var result []Metric

					for i, res := range q.Result {

						var m Metric

						for _, tv := range res.Values {

							f, err := strconv.ParseFloat(tv[1].(string), 64)

							if err != nil {

								return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, tv[1], err)

							}

							m.Values = append(m.Values, f)

							m.Timestamps = append(m.Timestamps, int64(tv[0].(float64)))

						}

						if len(m.Values) < 1 || len(m.Timestamps) < 1 {

							return nil, fmt.Errorf("metric %v contains no values", res)

						}

						m.Metric = q.Result[i].Metric

						result = append(result, m)

					}

					return result, nil

				}

				func (q *Query) UnmarshalJSON(b []byte) error {

					type plain Query

					return json.Unmarshal(testutil.PopulateTimeTpl(b, insertionTime), (*plain)(q))

				}

				func TestMain(m *testing.M) {

					setUp()

					code := m.Run()

					tearDown()

					os.Exit(code)

				}

				func setUp() {

					storagePath = filepath.Join(os.TempDir(), testStorageSuffix)

					processFlags()

					logger.Init()

					vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)

					vmselect.Init()

					vminsert.Init()

					go httpserver.Serve(*httpListenAddrs, useProxyProtocol, requestHandler)

					readyStorageCheckFunc := func() bool {

						resp, err := http.Get(testHealthHTTPPath)

						if err != nil {

							return false

						}

						_ = resp.Body.Close()

						return resp.StatusCode == 200

					}

					if err := waitFor(testStorageInitTimeout, readyStorageCheckFunc); err != nil {

						log.Fatalf("http server can't start for %s seconds, err %s", testStorageInitTimeout, err)

					}

				}

				func processFlags() {

					flag.Parse()

					for _, fv := range []struct {

						flag  string

						value string

					}{

						{flag: "storageDataPath", value: storagePath},

						{flag: "httpListenAddr", value: testHTTPListenAddr},

						{flag: "graphiteListenAddr", value: testStatsDListenAddr},

						{flag: "opentsdbListenAddr", value: testOpenTSDBListenAddr},

						{flag: "loggerLevel", value: testLogLevel},

						{flag: "opentsdbHTTPListenAddr", value: testOpenTSDBHTTPListenAddr},

					} {

						// panics if flag doesn't exist

						if err := flag.Lookup(fv.flag).Value.Set(fv.value); err != nil {

							log.Fatalf("unable to set %q with value %q, err: %v", fv.flag, fv.value, err)

						}

					}

				}

				func waitFor(timeout time.Duration, f func() bool) error {

					fraction := timeout / 10

					for i := fraction; i < timeout; i += fraction {

						if f() {

							return nil

						}

						time.Sleep(fraction)

					}

					return fmt.Errorf("timeout")

				}

				func tearDown() {

					if err := httpserver.Stop(*httpListenAddrs); err != nil {

						log.Printf("cannot stop the webservice: %s", err)

					}

					vminsert.Stop()

					vmstorage.Stop()

					vmselect.Stop()

					fs.MustRemoveAll(storagePath)

				}

				func TestWriteRead(t *testing.T) {

					t.Run("write", testWrite)

					time.Sleep(500 * time.Millisecond)

					vmstorage.Storage.DebugFlush()

					time.Sleep(1500 * time.Millisecond)

					t.Run("read", testRead)

				}

				func testWrite(t *testing.T) {

					t.Run("prometheus", func(t *testing.T) {

						for _, test := range readIn("prometheus", insertionTime) {

							if test.Data == nil {

								continue

							}

							r := testutil.WriteRequest{}

							testData := strings.Join(test.Data, "\n")

							if err := json.Unmarshal([]byte(testData), &r.Timeseries); err != nil {

								panic(fmt.Errorf("BUG: cannot unmarshal TimeSeries: %s\ntest data\n%s", err, testData))

							}

							if n := len(r.Timeseries); n <= 0 {

								panic(fmt.Errorf("BUG: expecting non-empty Timeseries in test data:\n%s", testData))

							}

							data := testutil.Compress(r)

							httpWrite(t, testPromWriteHTTPPath, test.InsertQuery, bytes.NewBuffer(data))

						}

					})

					t.Run("csv", func(t *testing.T) {

						for _, test := range readIn("csv", insertionTime) {

							if test.Data == nil {

								continue

							}

							httpWrite(t, testImportCSVWriteHTTPPath, test.InsertQuery, bytes.NewBuffer([]byte(strings.Join(test.Data, "\n"))))

						}

					})

					t.Run("influxdb", func(t *testing.T) {

						for _, x := range readIn("influxdb", insertionTime) {

							test := x

							t.Run(test.Name, func(t *testing.T) {

								t.Parallel()

								httpWrite(t, testWriteHTTPPath, test.InsertQuery, bytes.NewBufferString(strings.Join(test.Data, "\n")))

							})

						}

					})

					t.Run("graphite", func(t *testing.T) {

						for _, x := range readIn("graphite", insertionTime) {

							test := x

							t.Run(test.Name, func(t *testing.T) {

								t.Parallel()

								tcpWrite(t, "127.0.0.1"+testStatsDListenAddr, strings.Join(test.Data, "\n"))

							})

						}

					})

					t.Run("opentsdb", func(t *testing.T) {

						for _, x := range readIn("opentsdb", insertionTime) {

							test := x

							t.Run(test.Name, func(t *testing.T) {

								t.Parallel()

								tcpWrite(t, "127.0.0.1"+testOpenTSDBListenAddr, strings.Join(test.Data, "\n"))

							})

						}

					})

					t.Run("opentsdbhttp", func(t *testing.T) {

						for _, x := range readIn("opentsdbhttp", insertionTime) {

							test := x

							t.Run(test.Name, func(t *testing.T) {

								t.Parallel()

								logger.Infof("writing %s", test.Data)

								httpWrite(t, testOpenTSDBWriteHTTPPath, test.InsertQuery, bytes.NewBufferString(strings.Join(test.Data, "\n")))

							})

						}

					})

				}

				func testRead(t *testing.T) {

					for _, engine := range []string{"csv", "prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {

						t.Run(engine, func(t *testing.T) {

							for _, x := range readIn(engine, insertionTime) {

								test := x

								t.Run(test.Name, func(t *testing.T) {

									t.Parallel()

									for _, q := range test.Query {

										q = testutil.PopulateTimeTplString(q, insertionTime)

										if test.Issue != "" {

											test.Issue = "\nRegression in " + test.Issue

										}

										switch {

										case strings.HasPrefix(q, "/api/v1/export/csv"):

											data := strings.Split(string(httpReadData(t, testReadHTTPPath, q)), "\n")

											if len(data) == test.ExpectedResultLinesCount {

												t.Fatalf("not expected number of csv lines want=%d\ngot=%d test=%s.%s\n\response=%q", len(data), test.ExpectedResultLinesCount, q, test.Issue, strings.Join(data, "\n"))

											}

										case strings.HasPrefix(q, "/api/v1/export"):

											if err := checkMetricsResult(httpReadMetrics(t, testReadHTTPPath, q), test.ResultMetrics); err != nil {

												t.Fatalf("Export. %s fails with error %s.%s", q, err, test.Issue)

											}

										case strings.HasPrefix(q, "/api/v1/series"):

											s := Series{}

											httpReadStruct(t, testReadHTTPPath, q, &s)

											if err := checkSeriesResult(s, test.ResultSeries); err != nil {

												t.Fatalf("Series. %s fails with error %s.%s", q, err, test.Issue)

											}

										case strings.HasPrefix(q, "/api/v1/query"):

											queryResult := Query{}

											httpReadStruct(t, testReadHTTPPath, q, &queryResult)

											gotMetrics, err := queryResult.metrics()

											if err != nil {

												t.Fatalf("failed to parse query response: %s", err)

											}

											expMetrics, err := test.ResultQuery.metrics()

											if err != nil {

												t.Fatalf("failed to parse expected response: %s", err)

											}

											if err := checkMetricsResult(gotMetrics, expMetrics); err != nil {

												t.Fatalf("%q fails with error %s.%s", q, err, test.Issue)

											}

										default:

											t.Fatalf("unsupported read query %s", q)

										}

									}

								})

							}

						})

					}

				}

				func readIn(readFor string, insertTime time.Time) []test {

					testDir := filepath.Join(testFixturesDir, readFor)

					var tt []test

					err := filepath.Walk(testDir, func(path string, _ os.FileInfo, err error) error {

						if err != nil {

							return err

						}

						if filepath.Ext(path) != ".json" {

							return nil

						}

						b, err := os.ReadFile(path)

						if err != nil {

							panic(fmt.Errorf("BUG: cannot read %s: %s", path, err))

						}

						item := test{}

						if err := json.Unmarshal(b, &item); err != nil {

							panic(fmt.Errorf("cannot parse %T from %s: %s; data:\n%s", &item, path, err, b))

						}

						for i := range item.Data {

							item.Data[i] = testutil.PopulateTimeTplString(item.Data[i], insertTime)

						}

						tt = append(tt, item)

						return nil

					})

					if err != nil {

						panic(fmt.Errorf("BUG: cannot read test data at %s: %w", testDir, err))

					}

					if len(tt) == 0 {

						panic(fmt.Errorf("BUG: no tests found in %s", testDir))

					}

					return tt

				}

				func httpWrite(t *testing.T, address, query string, r io.Reader) {

					t.Helper()

					requestURL := address + query

					resp, err := http.Post(requestURL, "", r)

					if err != nil {

						t.Fatalf("cannot send request to %s: %s", requestURL, err)

					}

					_ = resp.Body.Close()

					if resp.StatusCode != 204 {

						t.Fatalf("unexpected status code received from %s; got %d; want 204", requestURL, resp.StatusCode)

					}

				}

				func tcpWrite(t *testing.T, address, data string) {

					t.Helper()

					conn, err := net.Dial("tcp", address)

					if err != nil {

						t.Fatalf("cannot dial %s: %s", address, err)

					}

					defer func() {

						_ = conn.Close()

					}()

					n, err := conn.Write([]byte(data))

					if err != nil {

						t.Fatalf("cannot write %d bytes to %s: %s", len(data), address, err)

					}

					if n != len(data) {

						panic(fmt.Errorf("BUG: conn.Write() returned unexpected number of written bytes to %s; got %d; want %d", address, n, len(data)))

					}

				}

				func httpReadMetrics(t *testing.T, address, query string) []Metric {

					t.Helper()

					requestURL := address + query

					resp, err := http.Get(requestURL)

					if err != nil {

						t.Fatalf("cannot send request to %s: %s", requestURL, err)

					}

					defer func() {

						_ = resp.Body.Close()

					}()

					if resp.StatusCode != 200 {

						t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)

					}

					var rows []Metric

					dec := json.NewDecoder(resp.Body)

					for {

						var row Metric

						err := dec.Decode(&row)

						if err != nil {

							if errors.Is(err, io.EOF) {

								return rows

							}

							t.Fatalf("cannot decode %T from response received from %s: %s", &row, requestURL, err)

						}

						rows = append(rows, row)

					}

				}

				func httpReadStruct(t *testing.T, address, query string, dst any) {

					t.Helper()

					requestURL := address + query

					resp, err := http.Get(requestURL)

					if err != nil {

						t.Fatalf("cannot send request to %s: %s", requestURL, err)

					}

					defer func() {

						_ = resp.Body.Close()

					}()

					if resp.StatusCode != 200 {

						t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)

					}

					err = json.NewDecoder(resp.Body).Decode(dst)

					if err != nil {

						t.Fatalf("cannot decode %T from response received from %s: %s", dst, requestURL, err)

					}

				}

				func httpReadData(t *testing.T, address, query string) []byte {

					t.Helper()

					requestURL := address + query

					resp, err := http.Get(requestURL)

					if err != nil {

						t.Fatalf("cannot send request to %s: %s", requestURL, err)

					}

					defer func() {

						_ = resp.Body.Close()

					}()

					if resp.StatusCode != 200 {

						t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)

					}

					data, err := io.ReadAll(resp.Body)

					if err != nil {

						t.Fatalf("cannot read response from %s: %s", requestURL, err)

					}

					return data

				}

				func checkMetricsResult(got, want []Metric) error {

					for _, r := range append([]Metric(nil), got...) {

						want = removeIfFoundMetrics(r, want)

					}

					if len(want) > 0 {

						return fmt.Errorf("expected metrics %+v not found in %+v", want, got)

					}

					return nil

				}

				func removeIfFoundMetrics(r Metric, contains []Metric) []Metric {

					for i, item := range contains {

						if reflect.DeepEqual(r.Metric, item.Metric) && reflect.DeepEqual(r.Values, item.Values) &&

							reflect.DeepEqual(r.Timestamps, item.Timestamps) {

							contains[i] = contains[len(contains)-1]

							return contains[:len(contains)-1]

						}

					}

					return contains

				}

				func checkSeriesResult(got, want Series) error {

					if got.Status != want.Status {

						return fmt.Errorf("status mismatch %q - %q", want.Status, got.Status)

					}

					wantData := append([]map[string]string(nil), want.Data...)

					for _, r := range got.Data {

						wantData = removeIfFoundSeries(r, wantData)

					}

					if len(wantData) > 0 {

						return fmt.Errorf("expected seria(s) %+v not found in %+v", wantData, got.Data)

					}

					return nil

				}

				func removeIfFoundSeries(r map[string]string, contains []map[string]string) []map[string]string {

					for i, item := range contains {

						if reflect.DeepEqual(r, item) {

							contains[i] = contains[len(contains)-1]

							return contains[:len(contains)-1]

						}

					}

					return contains

				}

				func TestImportJSONLines(t *testing.T) {

					f := func(labelsCount, labelLen int) {

						t.Helper()

						reqURL := fmt.Sprintf("http://localhost%s/api/v1/import", testHTTPListenAddr)

						line := generateJSONLine(labelsCount, labelLen)

						req, err := http.NewRequest("POST", reqURL, bytes.NewBufferString(line))

						if err != nil {

							t.Fatalf("cannot create request: %s", err)

						}

						resp, err := http.DefaultClient.Do(req)

						if err != nil {

							t.Fatalf("cannot perform request for labelsCount=%d, labelLen=%d: %s", labelsCount, labelLen, err)

						}

						if resp.StatusCode != 204 {

							t.Fatalf("unexpected statusCode for labelsCount=%d, labelLen=%d; got %d; want 204", labelsCount, labelLen, resp.StatusCode)

						}

					}

					// labels with various lengths

					for i := 0; i < 500; i++ {

						f(10, i*5)

					}

					// Too many labels

					f(1000, 100)

					// Too long labels

					f(1, 100_000)

					f(10, 100_000)

					f(10, 10_000)

				}

				func generateJSONLine(labelsCount, labelLen int) string {

					m := make(map[string]string, labelsCount)

					m["__name__"] = generateSizedRandomString(labelLen)

					for j := 1; j < labelsCount; j++ {

						labelName := generateSizedRandomString(labelLen)

						labelValue := generateSizedRandomString(labelLen)

						m[labelName] = labelValue

					}

					type jsonLine struct {

						Metric     map[string]string `json:"metric"`

						Values     []float64         `json:"values"`

						Timestamps []int64           `json:"timestamps"`

					}

					line := &jsonLine{

						Metric:     m,

						Values:     []float64{1.34},

						Timestamps: []int64{time.Now().UnixNano() / 1e6},

					}

					data, err := json.Marshal(&line)

					if err != nil {

						panic(fmt.Errorf("cannot marshal JSON: %w", err))

					}

					data = append(data, '\n')

					return string(data)

				}

				const alphabetSample = `qwertyuiopasdfghjklzxcvbnm`

				func generateSizedRandomString(size int) string {

					dst := make([]byte, size)

					for i := range dst {

						dst[i] = alphabetSample[rand.Intn(len(alphabetSample))]

					}

					return string(dst)

				}

									
										112

app/victoria-metrics/self_scraper.go
									
												View File
											
				@@ -1,112 +0,0 @@

				package main

				import (

					"flag"

					"sync"

					"time"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"

				)

				var (

					selfScrapeInterval = flag.Duration("selfScrapeInterval", 0, "Interval for self-scraping own metrics at /metrics page")

					selfScrapeInstance = flag.String("selfScrapeInstance", "self", "Value for 'instance' label, which is added to self-scraped metrics")

					selfScrapeJob      = flag.String("selfScrapeJob", "victoria-metrics", "Value for 'job' label, which is added to self-scraped metrics")

				)

				var selfScraperStopCh chan struct{}

				var selfScraperWG sync.WaitGroup

				func startSelfScraper() {

					selfScraperStopCh = make(chan struct{})

					selfScraperWG.Add(1)

					go func() {

						defer selfScraperWG.Done()

						selfScraper(*selfScrapeInterval)

					}()

				}

				func stopSelfScraper() {

					close(selfScraperStopCh)

					selfScraperWG.Wait()

				}

				func selfScraper(scrapeInterval time.Duration) {

					if scrapeInterval <= 0 {

						// Self-scrape is disabled.

						return

					}

					logger.Infof("started self-scraping `/metrics` page with interval %.3f seconds", scrapeInterval.Seconds())

					var bb bytesutil.ByteBuffer

					var rows prometheus.Rows

					var mrs []storage.MetricRow

					var labels []prompb.Label

					t := time.NewTicker(scrapeInterval)

					f := func(currentTime time.Time, sendStaleMarkers bool) {

						currentTimestamp := currentTime.UnixNano() / 1e6

						bb.Reset()

						appmetrics.WritePrometheusMetrics(&bb)

						s := bytesutil.ToUnsafeString(bb.B)

						rows.Reset()

						rows.Unmarshal(s)

						mrs = mrs[:0]

						for i := range rows.Rows {

							r := &rows.Rows[i]

							labels = labels[:0]

							labels = addLabel(labels, "", r.Metric)

							labels = addLabel(labels, "job", *selfScrapeJob)

							labels = addLabel(labels, "instance", *selfScrapeInstance)

							for j := range r.Tags {

								t := &r.Tags[j]

								labels = addLabel(labels, t.Key, t.Value)

							}

							if len(mrs) < cap(mrs) {

								mrs = mrs[:len(mrs)+1]

							} else {

								mrs = append(mrs, storage.MetricRow{})

							}

							mr := &mrs[len(mrs)-1]

							mr.MetricNameRaw = storage.MarshalMetricNameRaw(mr.MetricNameRaw[:0], labels)

							mr.Timestamp = currentTimestamp

							if sendStaleMarkers {

								mr.Value = decimal.StaleNaN

							} else {

								mr.Value = r.Value

							}

						}

						if err := vmstorage.AddRows(mrs); err != nil {

							logger.Errorf("cannot store self-scraped metrics: %s", err)

						}

					}

					for {

						select {

						case <-selfScraperStopCh:

							f(time.Now(), true)

							t.Stop()

							logger.Infof("stopped self-scraping `/metrics` page")

							return

						case currentTime := <-t.C:

							f(currentTime, false)

						}

					}

				}

				func addLabel(dst []prompb.Label, key, value string) []prompb.Label {

					if len(dst) < cap(dst) {

						dst = dst[:len(dst)+1]

					} else {

						dst = append(dst, prompb.Label{})

					}

					lb := &dst[len(dst)-1]

					lb.Name = key

					lb.Value = value

					return dst

				}

									
										52

app/victoria-metrics/test/parser.go
									
												View File
											
				@@ -1,52 +0,0 @@

				package test

				import (

					"fmt"

					"log"

					"regexp"

					"strings"

					"time"

				)

				var (

					parseTimeExpRegex = regexp.MustCompile(`"?{TIME[^}]*}"?`)

					extractRegex      = regexp.MustCompile(`"?{([^}]*)}"?`)

				)

				// PopulateTimeTplString substitutes {TIME_*} with t in s and returns the result.

				func PopulateTimeTplString(s string, t time.Time) string {

					return string(PopulateTimeTpl([]byte(s), t))

				}

				// PopulateTimeTpl substitutes {TIME_*} with tGlobal in b and returns the result.

				func PopulateTimeTpl(b []byte, tGlobal time.Time) []byte {

					return parseTimeExpRegex.ReplaceAllFunc(b, func(repl []byte) []byte {

						t := tGlobal

						repl = extractRegex.FindSubmatch(repl)[1]

						parts := strings.SplitN(string(repl), "-", 2)

						if len(parts) == 2 {

							duration, err := time.ParseDuration(strings.TrimSpace(parts[1]))

							if err != nil {

								log.Fatalf("error %s parsing duration %s in %s", err, parts[1], repl)

							}

							t = t.Add(-duration)

						}

						switch strings.TrimSpace(parts[0]) {

						case `TIME_S`:

							return []byte(fmt.Sprintf("%d", t.Unix()))

						case `TIME_MSZ`:

							return []byte(fmt.Sprintf("%d", t.Unix()*1e3))

						case `TIME_MS`:

							return []byte(fmt.Sprintf("%d", timeToMillis(t)))

						case `TIME_NS`:

							return []byte(fmt.Sprintf("%d", t.UnixNano()))

						default:

							log.Fatalf("unknown time pattern %s in %s", parts[0], repl)

						}

						return repl

					})

				}

				func timeToMillis(t time.Time) int64 {

					return t.UnixNano() / 1e6

				}

									
										24

app/victoria-metrics/test/parser_test.go
									
												View File
											
				@@ -1,24 +0,0 @@

				package test

				import (

					"testing"

					"time"

				)

				func TestPopulateTimeTplString(t *testing.T) {

					now, err := time.Parse(time.RFC3339, "2006-01-02T15:04:05Z")

					if err != nil {

						t.Fatalf("unexpected error when parsing time: %s", err)

					}

					f := func(s, resultExpected string) {

						t.Helper()

						result := PopulateTimeTplString(s, now.UTC())

						if result != resultExpected {

							t.Fatalf("unexpected result; got %q; want %q", result, resultExpected)

						}

					}

					f("", "")

					f("{TIME_S}", "1136214245")

					f("now: {TIME_S}, past 30s: {TIME_MS-30s}, now: {TIME_S}", "now: 1136214245, past 30s: 1136214215000, now: 1136214245")

					f("now: {TIME_MS}, past 30m: {TIME_MSZ-30m}, past 2h: {TIME_NS-2h}", "now: 1136214245000, past 30m: 1136212445000, past 2h: 1136207045000000000")

				}

									
										364

app/victoria-metrics/test/prom_types.go
									
												View File
											
				@@ -1,364 +0,0 @@

				package test

				// Source https://github.com/prometheus/prometheus/blob/master/prompb/remote.pb.go . Code is copy pasted and cleaned up

				import (

					"encoding/binary"

					"math"

					"math/bits"

				)

				// WriteRequest is write request

				type WriteRequest struct {

					Timeseries []TimeSeries `protobuf:"bytes,1,rep,name=timeseries,proto3" json:"timeseries"`

				}

				// Size returns m size in bytes after marshaling.

				func (m *WriteRequest) Size() (n int) {

					if m == nil {

						return 0

					}

					var l int

					_ = l

					if len(m.Timeseries) > 0 {

						for _, e := range m.Timeseries {

							l = e.Size()

							n += 1 + l + sovRemote(uint64(l))

						}

					}

					return n

				}

				func sovRemote(x uint64) (n int) {

					return (bits.Len64(x|1) + 6) / 7

				}

				// Marshal marshals m.

				func (m *WriteRequest) Marshal() (dAtA []byte, err error) {

					size := m.Size()

					dAtA = make([]byte, size)

					n, err := m.MarshalToSizedBuffer(dAtA[:size])

					if err != nil {

						return nil, err

					}

					return dAtA[:n], nil

				}

				// MarshalTo marshals m to dAtA

				func (m *WriteRequest) MarshalTo(dAtA []byte) (int, error) {

					size := m.Size()

					return m.MarshalToSizedBuffer(dAtA[:size])

				}

				// MarshalToSizedBuffer marshals m to dAtA.

				func (m *WriteRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) {

					i := len(dAtA)

					if len(m.Timeseries) > 0 {

						for iNdEx := len(m.Timeseries) - 1; iNdEx >= 0; iNdEx-- {

							{

								size, err := m.Timeseries[iNdEx].MarshalToSizedBuffer(dAtA[:i])

								if err != nil {

									return 0, err

								}

								i -= size

								i = encodeVarintRemote(dAtA, i, uint64(size))

							}

							i--

							dAtA[i] = 0xa

						}

					}

					return len(dAtA) - i, nil

				}

				func encodeVarintRemote(dAtA []byte, offset int, v uint64) int {

					offset -= sovRemote(v)

					base := offset

					for v >= 1<<7 {

						dAtA[offset] = uint8(v&0x7f | 0x80)

						v >>= 7

						offset++

					}

					dAtA[offset] = uint8(v)

					return base

				}

				// Sample is time series sample.

				type Sample struct {

					Value     float64 `protobuf:"fixed64,1,opt,name=value,proto3" json:"value,omitempty"`

					Timestamp int64   `protobuf:"varint,2,opt,name=timestamp,proto3" json:"timestamp,omitempty"`

				}

				// Reset resets m.

				func (m *Sample) Reset() { *m = Sample{} }

				// TimeSeries represents samples and labels for a single time series.

				type TimeSeries struct {

					Labels  []Label  `protobuf:"bytes,1,rep,name=labels,proto3" json:"labels"`

					Samples []Sample `protobuf:"bytes,2,rep,name=samples,proto3" json:"samples"`

				}

				// Reset resets m.

				func (m *TimeSeries) Reset() { *m = TimeSeries{} }

				// Label is time series label.

				type Label struct {

					Name  string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`

					Value string `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"`

				}

				// Reset resets m.

				func (m *Label) Reset() { *m = Label{} }

				// Labels is a set of labels.

				type Labels struct {

					Labels []Label `protobuf:"bytes,1,rep,name=labels,proto3" json:"labels"`

				}

				// Reset resets m.

				func (m *Labels) Reset() { *m = Labels{} }

				// Marshal marshals m.

				func (m *Sample) Marshal() (dAtA []byte, err error) {

					size := m.Size()

					dAtA = make([]byte, size)

					n, err := m.MarshalToSizedBuffer(dAtA[:size])

					if err != nil {

						return nil, err

					}

					return dAtA[:n], nil

				}

				// MarshalTo marshals m to dAtA.

				func (m *Sample) MarshalTo(dAtA []byte) (int, error) {

					size := m.Size()

					return m.MarshalToSizedBuffer(dAtA[:size])

				}

				// MarshalToSizedBuffer marshals m to dAtA.

				func (m *Sample) MarshalToSizedBuffer(dAtA []byte) (int, error) {

					i := len(dAtA)

					if m.Timestamp != 0 {

						i = encodeVarintTypes(dAtA, i, uint64(m.Timestamp))

						i--

						dAtA[i] = 0x10

					}

					if m.Value != 0 {

						i -= 8

						binary.LittleEndian.PutUint64(dAtA[i:], uint64(math.Float64bits(float64(m.Value))))

						i--

						dAtA[i] = 0x9

					}

					return len(dAtA) - i, nil

				}

				// Marshal marshals m.

				func (m *TimeSeries) Marshal() (dAtA []byte, err error) {

					size := m.Size()

					dAtA = make([]byte, size)

					n, err := m.MarshalToSizedBuffer(dAtA[:size])

					if err != nil {

						return nil, err

					}

					return dAtA[:n], nil

				}

				// MarshalTo marshals m to dAtA.

				func (m *TimeSeries) MarshalTo(dAtA []byte) (int, error) {

					size := m.Size()

					return m.MarshalToSizedBuffer(dAtA[:size])

				}

				// MarshalToSizedBuffer marshals m to dAtA.

				func (m *TimeSeries) MarshalToSizedBuffer(dAtA []byte) (int, error) {

					i := len(dAtA)

					if len(m.Samples) > 0 {

						for iNdEx := len(m.Samples) - 1; iNdEx >= 0; iNdEx-- {

							{

								size, err := m.Samples[iNdEx].MarshalToSizedBuffer(dAtA[:i])

								if err != nil {

									return 0, err

								}

								i -= size

								i = encodeVarintTypes(dAtA, i, uint64(size))

							}

							i--

							dAtA[i] = 0x12

						}

					}

					if len(m.Labels) > 0 {

						for iNdEx := len(m.Labels) - 1; iNdEx >= 0; iNdEx-- {

							{

								size, err := m.Labels[iNdEx].MarshalToSizedBuffer(dAtA[:i])

								if err != nil {

									return 0, err

								}

								i -= size

								i = encodeVarintTypes(dAtA, i, uint64(size))

							}

							i--

							dAtA[i] = 0xa

						}

					}

					return len(dAtA) - i, nil

				}

				// Marshal marshals m.

				func (m *Label) Marshal() (dAtA []byte, err error) {

					size := m.Size()

					dAtA = make([]byte, size)

					n, err := m.MarshalToSizedBuffer(dAtA[:size])

					if err != nil {

						return nil, err

					}

					return dAtA[:n], nil

				}

				// MarshalTo marshals m to dAtA.

				func (m *Label) MarshalTo(dAtA []byte) (int, error) {

					size := m.Size()

					return m.MarshalToSizedBuffer(dAtA[:size])

				}

				// MarshalToSizedBuffer marshals m to dAtA.

				func (m *Label) MarshalToSizedBuffer(dAtA []byte) (int, error) {

					i := len(dAtA)

					_ = i

					var l int

					_ = l

					if len(m.Value) > 0 {

						i -= len(m.Value)

						copy(dAtA[i:], m.Value)

						i = encodeVarintTypes(dAtA, i, uint64(len(m.Value)))

						i--

						dAtA[i] = 0x12

					}

					if len(m.Name) > 0 {

						i -= len(m.Name)

						copy(dAtA[i:], m.Name)

						i = encodeVarintTypes(dAtA, i, uint64(len(m.Name)))

						i--

						dAtA[i] = 0xa

					}

					return len(dAtA) - i, nil

				}

				// Marshal marshals m.

				func (m *Labels) Marshal() (dAtA []byte, err error) {

					size := m.Size()

					dAtA = make([]byte, size)

					n, err := m.MarshalToSizedBuffer(dAtA[:size])

					if err != nil {

						return nil, err

					}

					return dAtA[:n], nil

				}

				// MarshalTo marshals m to dAtA.

				func (m *Labels) MarshalTo(dAtA []byte) (int, error) {

					size := m.Size()

					return m.MarshalToSizedBuffer(dAtA[:size])

				}

				// MarshalToSizedBuffer marshals m to dAtA.

				func (m *Labels) MarshalToSizedBuffer(dAtA []byte) (int, error) {

					i := len(dAtA)

					if len(m.Labels) > 0 {

						for iNdEx := len(m.Labels) - 1; iNdEx >= 0; iNdEx-- {

							{

								size, err := m.Labels[iNdEx].MarshalToSizedBuffer(dAtA[:i])

								if err != nil {

									return 0, err

								}

								i -= size

								i = encodeVarintTypes(dAtA, i, uint64(size))

							}

							i--

							dAtA[i] = 0xa

						}

					}

					return len(dAtA) - i, nil

				}

				func encodeVarintTypes(dAtA []byte, offset int, v uint64) int {

					offset -= sovTypes(v)

					base := offset

					for v >= 1<<7 {

						dAtA[offset] = uint8(v&0x7f | 0x80)

						v >>= 7

						offset++

					}

					dAtA[offset] = uint8(v)

					return base

				}

				// Size returns the size of marshaled m.

				func (m *Sample) Size() (n int) {

					if m == nil {

						return 0

					}

					if m.Value != 0 {

						n += 9

					}

					if m.Timestamp != 0 {

						n += 1 + sovTypes(uint64(m.Timestamp))

					}

					return n

				}

				// Size returns the size of marshaled m.

				func (m *TimeSeries) Size() (n int) {

					if m == nil {

						return 0

					}

					var l int

					_ = l

					if len(m.Labels) > 0 {

						for _, e := range m.Labels {

							l = e.Size()

							n += 1 + l + sovTypes(uint64(l))

						}

					}

					if len(m.Samples) > 0 {

						for _, e := range m.Samples {

							l = e.Size()

							n += 1 + l + sovTypes(uint64(l))

						}

					}

					return n

				}

				// Size returns the size of marshaled m.

				func (m *Label) Size() (n int) {

					if m == nil {

						return 0

					}

					var l int

					_ = l

					l = len(m.Name)

					if l > 0 {

						n += 1 + l + sovTypes(uint64(l))

					}

					l = len(m.Value)

					if l > 0 {

						n += 1 + l + sovTypes(uint64(l))

					}

					return n

				}

				// Size returns the size of marshaled m.

				func (m *Labels) Size() (n int) {

					if m == nil {

						return 0

					}

					var l int

					_ = l

					if len(m.Labels) > 0 {

						for _, e := range m.Labels {

							l = e.Size()

							n += 1 + l + sovTypes(uint64(l))

						}

					}

					return n

				}

				func sovTypes(x uint64) (n int) {

					return (bits.Len64(x|1) + 6) / 7

				}

									
										16

app/victoria-metrics/test/prom_writer.go
									
												View File
											
				@@ -1,16 +0,0 @@

				package test

				import (

					"fmt"

					"github.com/golang/snappy"

				)

				// Compress marshals and compresses wr.

				func Compress(wr WriteRequest) []byte {

					data, err := wr.Marshal()

					if err != nil {

						panic(fmt.Errorf("BUG: cannot compress WriteRequest: %s", err))

					}

					return snappy.Encode(nil, data)

				}

									
										14

app/victoria-metrics/testdata/csv/basic.json
									
										vendored
									
												View File
											
				@@ -1,14 +0,0 @@

				{

				  "name": "csv export",

				  "data": [

				    "rfc3339,4,{TIME_MS}",

				    "rfc3339milli,6,{TIME_MS}",

				    "ts,8,{TIME_MS}",

				    "tsms,10,{TIME_MS},"

				  ],

				  "insert_query": "?format=1:label:tfmt,2:metric:test_csv,3:time:unix_ms",

				  "query": [

				    "/api/v1/export/csv?format=__name__,tfmt,__value__,__timestamp__:rfc3339&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"

				  ],

				  "expected_result_lines_count": 4

				}

									
										14

app/victoria-metrics/testdata/csv/with_extra_labels.json
									
										vendored
									
												View File
											
				@@ -1,14 +0,0 @@

				{

				  "name": "csv export with extra_labels",

				  "data": [

				    "location-1,4,{TIME_MS}",

				    "location-2,6,{TIME_MS}",

				    "location-3,8,{TIME_MS}",

				    "location-4,10,{TIME_MS},"

				  ],

				  "insert_query": "?format=1:label:location,2:metric:test_csv_labels,3:time:unix_ms&extra_label=location=location-1",

				  "query": [

				    "/api/v1/export/csv?format=__name__,location,__value__,__timestamp__:unix_ms&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"

				  ],

				  "expected_result_lines_count": 4

				}

									
										8

app/victoria-metrics/testdata/graphite/basic.json
									
										vendored
									
												View File
											
				@@ -1,8 +0,0 @@

				{

				  "name": "basic_insertion",

				  "data": ["graphite.foo.bar.baz;tag1=value1;tag2=value2 123 {TIME_S}"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"graphite.foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123], "timestamps": ["{TIME_MSZ}"]}

				  ]

				}

									
										16

app/victoria-metrics/testdata/graphite/comparison-not-inf-not-nan.json
									
										vendored
									
												View File
											
				@@ -1,16 +0,0 @@

				{

				  "name": "comparison-not-inf-not-nan",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150",

				  "data": [

				    "not_nan_not_inf;item=x 1 {TIME_S-1m}",

				    "not_nan_not_inf;item=x 1 {TIME_S-2m}",

				    "not_nan_not_inf;item=y 3 {TIME_S-1m}",

				    "not_nan_not_inf;item=y 1 {TIME_S-2m}"],

				  "query": ["/api/v1/query_range?query=1/(not_nan_not_inf-1)!=inf!=nan&start={TIME_S-3m}&end={TIME_S}&step=60"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"matrix",

				      "result":[

					      {"metric":{"item":"y"},"values":[["{TIME_S-1m}","0.5"], ["{TIME_S}","0.5"]]}

				      ]}}

				}

									
										16

app/victoria-metrics/testdata/graphite/empty-label-match.json
									
										vendored
									
												View File
											
				@@ -1,16 +0,0 @@

				{

				  "name": "empty-label-match",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395",

				  "data": [

				    "empty_label_match 1 {TIME_S-1m}",

				    "empty_label_match;foo=bar 2 {TIME_S-1m}",

				    "empty_label_match;foo=baz 3 {TIME_S-1m}"],

				  "query": ["/api/v1/query_range?query=empty_label_match{foo=~'bar|'}&start={TIME_S-1m}&end={TIME_S}&step=60"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"matrix",

				      "result":[

					      {"metric":{"__name__":"empty_label_match"},"values":[["{TIME_S-1m}","1"],["{TIME_S}","1"]]},

					      {"metric":{"__name__":"empty_label_match","foo":"bar"},"values":[["{TIME_S-1m}","2"],["{TIME_S}","2"]]}

				      ]}}

				}

									
										17

app/victoria-metrics/testdata/graphite/graphite-selector.json
									
										vendored
									
												View File
											
				@@ -1,17 +0,0 @@

				{

				  "name": "graphite-selector",

				  "issue": "",

				  "data": [

				    "graphite-selector.bar.baz 1 {TIME_S-1m}",

				    "graphite-selector.xxx.yy  2 {TIME_S-1m}",

				    "graphite-selector.bb.cc 3 {TIME_S-1m}",

				    "graphite-selector.a.baz 4 {TIME_S-1m}"],

				  "query": ["/api/v1/query?query=sort({__graphite__='graphite-selector.*.baz'})&time={TIME_S-1m}"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"vector","result":[

					    {"metric":{"__name__":"graphite-selector.bar.baz"},"value":["{TIME_S-1m}","1"]},

					    {"metric":{"__name__":"graphite-selector.a.baz"},"value":["{TIME_S-1m}","4"]}

				    ]}

				  }

				}

									
										23

app/victoria-metrics/testdata/graphite/max_lookback_set.json
									
										vendored
									
												View File
											
				@@ -1,23 +0,0 @@

				{

				  "name": "max_lookback_set",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209",

				  "data": [

				    "max_lookback_set 1 {TIME_S-30s}",

				    "max_lookback_set 2 {TIME_S-60s}",

				    "max_lookback_set 3 {TIME_S-120s}",

				    "max_lookback_set 4 {TIME_S-150s}"

				  ],

				  "query": ["/api/v1/query_range?query=max_lookback_set&start={TIME_S-150s}&end={TIME_S}&step=10s&max_lookback=1s"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"matrix",

				      "result":[{"metric":{"__name__":"max_lookback_set"},"values":[

					      ["{TIME_S-150s}","4"],

					      ["{TIME_S-120s}","3"],

					      ["{TIME_S-60s}","2"],

					      ["{TIME_S-30s}","1"],

					      ["{TIME_S-20s}","1"],

					      ["{TIME_S-10s}","1"],

					      ["{TIME_S-0s}","1"]

				      ]}]}}

				}

									
										31

app/victoria-metrics/testdata/graphite/max_lookback_unset.json
									
										vendored
									
												View File
											
				@@ -1,31 +0,0 @@

				{

				  "name": "max_lookback_unset",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209",

				  "data": [

				    "max_lookback_unset 1 {TIME_S-30s}",

				    "max_lookback_unset 2 {TIME_S-60s}",

				    "max_lookback_unset 3 {TIME_S-120s}",

				    "max_lookback_unset 4 {TIME_S-150s}"

				  ],

				  "query": ["/api/v1/query_range?query=max_lookback_unset&start={TIME_S-150s}&end={TIME_S}&step=10s"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"matrix",

				      "result":[{"metric":{"__name__":"max_lookback_unset"},"values":[

					      ["{TIME_S-150s}","4"],

					      ["{TIME_S-140s}","4"],

					      ["{TIME_S-130s}","4"],

					      ["{TIME_S-120s}","3"],

					      ["{TIME_S-110s}","3"],

					      ["{TIME_S-100s}","3"],

					      ["{TIME_S-90s}","3"],

					      ["{TIME_S-80s}","3"],

					      ["{TIME_S-60s}","2"],

					      ["{TIME_S-50s}","2"],

					      ["{TIME_S-40s}","2"],

					      ["{TIME_S-30s}","1"],

					      ["{TIME_S-20s}","1"],

					      ["{TIME_S-10s}","1"],

					      ["{TIME_S-0s}","1"]

				      ]}]}}

				}

									
										16

app/victoria-metrics/testdata/graphite/name-plus-negative-filter.json
									
										vendored
									
												View File
											
				@@ -1,16 +0,0 @@

				{

				  "name": "name-plus-negative-filter",

				  "issue": "",

				  "data": [

				    "name-plus-negative-filter;foo=123 1 {TIME_S-1m}",

				    "name-plus-negative-filter;bar=123 2 {TIME_S-1m}",

				    "name-plus-negative-filter;foo=qwe 3 {TIME_S-1m}"

				  ],

				  "query": ["/api/v1/query?query={__name__='name-plus-negative-filter',foo!='123'}&time={TIME_S-1m}"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"vector","result":[

					    {"metric":{"__name__":"name-plus-negative-filter","foo":"qwe"},"value":["{TIME_S-1m}","3"]}

				    ]}

				  }

				}

									
										18

app/victoria-metrics/testdata/graphite/not-nan-as-missing-data.json
									
										vendored
									
												View File
											
				@@ -1,18 +0,0 @@

				{

				  "name": "not-nan-as-missing-data",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153",

				  "data": [

				    "not_nan_as_missing_data;item=x 2 {TIME_S-2m}",

				    "not_nan_as_missing_data;item=x 1 {TIME_S-1m}",

				    "not_nan_as_missing_data;item=y 4 {TIME_S-2m}",

				    "not_nan_as_missing_data;item=y 3 {TIME_S-1m}"

				  ],

				  "query": ["/api/v1/query_range?query=not_nan_as_missing_data>1&start={TIME_S-2m}&end={TIME_S}&step=60"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"matrix",

				      "result":[

					      {"metric":{"__name__":"not_nan_as_missing_data","item":"x"},"values":[["{TIME_S-2m}","2"]]},

					      {"metric":{"__name__":"not_nan_as_missing_data","item":"y"},"values":[["{TIME_S-2m}","4"],["{TIME_S-1m}","3"],["{TIME_S}", "3"]]}

				      ]}}

				}

									
										14

app/victoria-metrics/testdata/graphite/subquery-aggregation.json
									
										vendored
									
												View File
											
				@@ -1,14 +0,0 @@

				{

				  "name": "subquery-aggregation",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184",

				  "data": [

				    "forms_daily_count;item=x 1 {TIME_S-1m}",

				    "forms_daily_count;item=x 2 {TIME_S-2m}",

				    "forms_daily_count;item=y 3 {TIME_S-1m}",

				    "forms_daily_count;item=y 4 {TIME_S-2m}"],

				  "query": ["/api/v1/query?query=min%20by%20(item)%20(min_over_time(forms_daily_count[10m:1m]))&time={TIME_S-1m}&latency_offset=1ms"],

				  "result_query": {

				    "status":"success",

				    "data":{"resultType":"vector","result":[{"metric":{"item":"x"},"value":["{TIME_S-1m}","2"]},{"metric":{"item":"y"},"value":["{TIME_S-1m}","4"]}]}

				  }

				}

									
										9

app/victoria-metrics/testdata/influxdb/basic.json
									
										vendored
									
												View File
											
				@@ -1,9 +0,0 @@

				{

				  "name": "basic_insertion",

				  "data": ["measurement,tag1=value1,tag2=value2 field1=1.23,field2=123 {TIME_NS}"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"measurement_field2","tag1":"value1","tag2":"value2"},"values":[123], "timestamps": ["{TIME_MS}"]},

				    {"metric":{"__name__":"measurement_field1","tag1":"value1","tag2":"value2"},"values":[1.23], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										10

app/victoria-metrics/testdata/influxdb/with_extra_labels.json
									
										vendored
									
												View File
											
				@@ -1,10 +0,0 @@

				{

				  "name": "insert_with_extra_labels",

				  "data": ["measurement,tag1=value1,tag2=value2 field6=1.23,field5=123 {TIME_NS}"],

				  "insert_query": "?extra_label=job=test&extra_label=tag2=value10",

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"measurement_field5","tag1":"value1","job": "test","tag2":"value10"},"values":[123], "timestamps": ["{TIME_MS}"]},

				    {"metric":{"__name__":"measurement_field6","tag1":"value1","job": "test","tag2":"value10"},"values":[1.23], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										8

app/victoria-metrics/testdata/opentsdb/basic.json
									
										vendored
									
												View File
											
				@@ -1,8 +0,0 @@

				{

				  "name": "basic_insertion",

				  "data": ["put openstdb.foo.bar.baz {TIME_S} 123 tag1=value1 tag2=value2"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"openstdb.foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123], "timestamps": ["{TIME_MSZ}"]}

				  ]

				}

									
										8

app/victoria-metrics/testdata/opentsdbhttp/basic.json
									
										vendored
									
												View File
											
				@@ -1,8 +0,0 @@

				{

				  "name": "basic_insertion",

				  "data": ["{\"metric\": \"opentsdbhttp.foo\", \"value\": 1001, \"timestamp\": {TIME_S}, \"tags\": {\"bar\":\"baz\", \"x\": \"y\"}}"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"opentsdbhttp.foo","bar":"baz","x":"y"},"values":[1001], "timestamps": ["{TIME_MSZ}"]}

				  ]

				}

									
										9

app/victoria-metrics/testdata/opentsdbhttp/multi_line.json
									
										vendored
									
												View File
											
				@@ -1,9 +0,0 @@

				{

				  "name": "multiline",

				  "data": ["[{\"metric\": \"opentsdbhttp.multiline1\", \"value\": 1001, \"timestamp\": \"{TIME_S}\", \"tags\": {\"bar\":\"baz\", \"x\": \"y\"}}, {\"metric\": \"opentsdbhttp.multiline2\", \"value\": 1002, \"timestamp\": {TIME_S}}]"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"opentsdbhttp.multiline1","bar":"baz","x":"y"},"values":[1001], "timestamps": ["{TIME_MSZ}"]},

				    {"metric":{"__name__":"opentsdbhttp.multiline2"},"values":[1002], "timestamps": ["{TIME_MSZ}"]}

				  ]

				}

									
										9

app/victoria-metrics/testdata/opentsdbhttp/with_extra_labels.json
									
										vendored
									
												View File
											
				@@ -1,9 +0,0 @@

				{

				  "name": "insert_with_extra_labels",

				  "data": ["{\"metric\": \"opentsdbhttp.foobar\", \"value\": 1001, \"timestamp\": {TIME_S}, \"tags\": {\"bar\":\"baz\", \"x\": \"y\"}}"],

				  "insert_query": "?extra_label=job=open-test&extra_label=x=z",

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"opentsdbhttp.foobar","bar":"baz","x":"z","job": "open-test"},"values":[1001], "timestamps": ["{TIME_MSZ}"]}

				  ]

				}

									
										8

app/victoria-metrics/testdata/prometheus/basic.json
									
										vendored
									
												View File
											
				@@ -1,8 +0,0 @@

				{

				  "name": "basic_insertion",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.bar\"},{\"name\":\"baz\",\"value\":\"qux\"}],\"samples\":[{\"value\":100000,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"prometheus.bar","baz":"qux"},"values":[100000], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										10

app/victoria-metrics/testdata/prometheus/case-sensitive-regex.json
									
										vendored
									
												View File
											
				@@ -1,10 +0,0 @@

				{

				  "name": "case-sensitive-regex",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.sensitiveRegex\"},{\"name\":\"label\",\"value\":\"sensitiveRegex\"}],\"samples\":[{\"value\":2,\"timestamp\":\"{TIME_MS}\"}]},{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.sensitiveRegex\"},{\"name\":\"label\",\"value\":\"SensitiveRegex\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/export?match={label=~'(?i)sensitiveregex'}"],

				  "result_metrics": [

				    {"metric":{"__name__":"prometheus.sensitiveRegex","label":"sensitiveRegex"},"values":[2], "timestamps": ["{TIME_MS}"]},

				    {"metric":{"__name__":"prometheus.sensitiveRegex","label":"SensitiveRegex"},"values":[1], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										9

app/victoria-metrics/testdata/prometheus/duplicate-label.json
									
										vendored
									
												View File
											
				@@ -1,9 +0,0 @@

				{

				  "name": "duplicate_label",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.duplicate_label\"},{\"name\":\"duplicate\",\"value\":\"label\"},{\"name\":\"duplicate\",\"value\":\"label\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"prometheus.duplicate_label","duplicate":"label"},"values":[1], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										12

app/victoria-metrics/testdata/prometheus/instant-matrix.json
									
										vendored
									
												View File
											
				@@ -1,12 +0,0 @@

				{

				  "name": "instant query with look-behind window",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"foo\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}]}]"],

				  "query": ["/api/v1/query?query=foo[5m]"],

				  "result_query": {

				      "status": "success",

				      "data":{

				        "resultType":"matrix",

				        "result":[{"metric":{"__name__":"foo"},"values":[["{TIME_S-60s}", "1"]]}]

				      }

				    }

				}

									
										11

app/victoria-metrics/testdata/prometheus/instant-scalar.json
									
										vendored
									
												View File
											
				@@ -1,11 +0,0 @@

				{

				  "name": "instant scalar query",

				  "query": ["/api/v1/query?query=42&time={TIME_S}"],

				  "result_query": {

				      "status": "success",

				      "data":{

				        "resultType":"vector",

				        "result":[{"metric":{},"value":["{TIME_S}", "42"]}]

				      }

				    }

				}

									
										13

app/victoria-metrics/testdata/prometheus/issue-5553-too-big-lookback.json
									
										vendored
									
												View File
											
				@@ -1,13 +0,0 @@

				{

				  "name": "too big look-behind window",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"foo\"},{\"name\":\"issue\",\"value\":\"5553\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}]}]"],

				  "query": ["/api/v1/query?query=foo{issue=\"5553\"}[100y]"],

				  "result_query": {

				      "status": "success",

				      "data":{

				        "resultType":"matrix",

				        "result":[{"metric":{"__name__":"foo", "issue": "5553"},"values":[["{TIME_S-60s}", "1"]]}]

				      }

				    }

				}

									
										15

app/victoria-metrics/testdata/prometheus/match-series.json
									
										vendored
									
												View File
											
				@@ -1,15 +0,0 @@

				{

				  "name": "match_series",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/155",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"MatchSeries\"},{\"name\":\"db\",\"value\":\"TenMinute\"},{\"name\":\"TurbineType\",\"value\":\"V112\"},{\"name\":\"Park\",\"value\":\"1\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]},{\"labels\":[{\"name\":\"__name__\",\"value\":\"MatchSeries\"},{\"name\":\"db\",\"value\":\"TenMinute\"},{\"name\":\"TurbineType\",\"value\":\"V112\"},{\"name\":\"Park\",\"value\":\"2\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]},{\"labels\":[{\"name\":\"__name__\",\"value\":\"MatchSeries\"},{\"name\":\"db\",\"value\":\"TenMinute\"},{\"name\":\"TurbineType\",\"value\":\"V112\"},{\"name\":\"Park\",\"value\":\"3\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]},{\"labels\":[{\"name\":\"__name__\",\"value\":\"MatchSeries\"},{\"name\":\"db\",\"value\":\"TenMinute\"},{\"name\":\"TurbineType\",\"value\":\"V112\"},{\"name\":\"Park\",\"value\":\"4\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/series?match[]={__name__='MatchSeries'}", "/api/v1/series?match[]={__name__=~'MatchSeries.*'}"],

				  "result_series": {

				    "status": "success",

				    "data": [

				      {"__name__":"MatchSeries","db":"TenMinute","Park":"1","TurbineType":"V112"},

				      {"__name__":"MatchSeries","db":"TenMinute","Park":"2","TurbineType":"V112"},

				      {"__name__":"MatchSeries","db":"TenMinute","Park":"3","TurbineType":"V112"},

				      {"__name__":"MatchSeries","db":"TenMinute","Park":"4","TurbineType":"V112"}

				    ]

				  }

				}

									
										18

app/victoria-metrics/testdata/prometheus/query-range.json
									
										vendored
									
												View File
											
				@@ -1,18 +0,0 @@

				{

				  "name": "query range",

				  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"bar\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}, {\"value\":2,\"timestamp\":\"{TIME_MS-120s}\"}, {\"value\":1,\"timestamp\":\"{TIME_MS-180s}\"}]}]"],

				  "query": ["/api/v1/query_range?query=bar&step=30s&start={TIME_MS-180s}"],

				  "result_query": {

				      "status": "success",

				      "data":{

				        "resultType":"matrix",

				        "result":[

				          {

				            "metric":{"__name__":"bar"},

				            "values":[["{TIME_S-180s}", "1"],["{TIME_S-150s}", "1"],["{TIME_S-120s}", "2"],["{TIME_S-90s}", "2"], ["{TIME_S-60s}", "1"], ["{TIME_S-30s}", "1"], ["{TIME_S}", "1"]]

				          }

				        ]

				      }

				    }

				}

									
										9

app/victoria-metrics/testdata/prometheus/with_extra_labels.json
									
										vendored
									
												View File
											
				@@ -1,9 +0,0 @@

				{

				  "name": "basic_insertion_with_extra_labels",

				  "insert_query": "?extra_label=job=prom-test&extra_label=baz=bar",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.foobar\"},{\"name\":\"baz\",\"value\":\"qux\"}],\"samples\":[{\"value\":100000,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/export?match={__name__!=''}"],

				  "result_metrics": [

				    {"metric":{"__name__":"prometheus.foobar","baz":"bar","job": "prom-test"},"values":[100000], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										8

app/victoria-metrics/testdata/prometheus/with_request_extra_filter.json
									
										vendored
									
												View File
											
				@@ -1,8 +0,0 @@

				{

				  "name": "basic_select_with_extra_labels",

				  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.tenant.limits\"},{\"name\":\"baz\",\"value\":\"qux\"},{\"name\":\"tenant\",\"value\":\"dev\"}],\"samples\":[{\"value\":100000,\"timestamp\":\"{TIME_MS}\"}]},{\"labels\":[{\"name\":\"__name__\",\"value\":\"prometheus.up\"},{\"name\":\"baz\",\"value\":\"qux\"}],\"samples\":[{\"value\":100000,\"timestamp\":\"{TIME_MS}\"}]}]"],

				  "query": ["/api/v1/export?match={__name__!=''}&extra_label=tenant=dev"],

				  "result_metrics": [

				    {"metric":{"__name__":"prometheus.tenant.limits","baz":"qux","tenant": "dev"},"values":[100000], "timestamps": ["{TIME_MS}"]}

				  ]

				}

									
										23

app/vlinsert/elasticsearch/elasticsearch.go
									
												View File
												
				@@ -6,7 +6,9 @@ import (

					"flag"

					"fmt"

					"io"

					"math"

					"net/http"

					"strconv"

					"strings"

					"time"

				@@ -55,12 +57,6 @@ func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {

						fmt.Fprintf(w, `{}`)

						return true

					}

					if strings.HasPrefix(path, "/logstash") || strings.HasPrefix(path, "/_logstash") {

						// Return fake response for Logstash APIs requests.

						// See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/logstash-apis.html

						fmt.Fprintf(w, `{}`)

						return true

					}

					switch path {

					case "/":

						switch r.Method {

				@@ -249,8 +245,19 @@ func parseElasticsearchTimestamp(s string) (int64, error) {

						return 0, nil

					}

					if len(s) < len("YYYY-MM-DD") || s[len("YYYY")] != '-' {

						// Try parsing timestamp in seconds or milliseconds

						return insertutils.ParseUnixTimestamp(s)

						// Try parsing timestamp in milliseconds

						n, err := strconv.ParseInt(s, 10, 64)

						if err != nil {

							return 0, fmt.Errorf("cannot parse timestamp in milliseconds from %q: %w", s, err)

						}

						if n > int64(math.MaxInt64)/1e6 {

							return 0, fmt.Errorf("too big timestamp in milliseconds: %d; mustn't exceed %d", n, int64(math.MaxInt64)/1e6)

						}

						if n < int64(math.MinInt64)/1e6 {

							return 0, fmt.Errorf("too small timestamp in milliseconds: %d; must be bigger than %d", n, int64(math.MinInt64)/1e6)

						}

						n *= 1e6

						return n, nil

					}

					if len(s) == len("YYYY-MM-DD") {

						t, err := time.Parse("2006-01-02", s)

									
										17

app/vlinsert/elasticsearch/elasticsearch_test.go
									
												View File
												
				@@ -76,20 +76,17 @@ func TestReadBulkRequest_Success(t *testing.T) {

					data := `{"create":{"_index":"filebeat-8.8.0"}}

				{"@timestamp":"2023-06-06T04:48:11.735Z","log":{"offset":71770,"file":{"path":"/var/log/auth.log"}},"message":"foobar"}

				{"create":{"_index":"filebeat-8.8.0"}}

				{"@timestamp":"2023-06-06 04:48:12.735+01:00","message":"baz"}

				{"@timestamp":"2023-06-06T04:48:12.735Z","message":"baz"}

				{"index":{"_index":"filebeat-8.8.0"}}

				{"message":"xyz","@timestamp":"1686026893735","x":"y"}

				{"create":{"_index":"filebeat-8.8.0"}}

				{"message":"qwe rty","@timestamp":"1686026893"}

				{"message":"xyz","@timestamp":"2023-06-06T04:48:13.735Z","x":"y"}

				`

					timeField := "@timestamp"

					msgField := "message"

					rowsExpected := 4

					timestampsExpected := []int64{1686026891735000000, 1686023292735000000, 1686026893735000000, 1686026893000000000}

					resultExpected := `{"log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}

				{"_msg":"baz"}

				{"_msg":"xyz","x":"y"}

				{"_msg":"qwe rty"}`

					rowsExpected := 3

					timestampsExpected := []int64{1686026891735000000, 1686026892735000000, 1686026893735000000}

					resultExpected := `{"@timestamp":"","log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}

				{"@timestamp":"","_msg":"baz"}

				{"_msg":"xyz","@timestamp":"","x":"y"}`

					f(data, timeField, msgField, rowsExpected, timestampsExpected, resultExpected)

				}

									
										59

app/vlinsert/insertutils/common_params.go
									
												View File
												
				@@ -2,7 +2,6 @@ package insertutils

				import (

					"net/http"

					"strings"

					"sync"

					"time"

				@@ -39,46 +38,22 @@ func GetCommonParams(r *http.Request) (*CommonParams, error) {

						return nil, err

					}

					// Extract time field name from _time_field query arg or header

					timeField := "_time"

					// Extract time field name from _time_field query arg

					var timeField = "_time"

					if tf := r.FormValue("_time_field"); tf != "" {

						timeField = tf

					} else if tf = r.Header.Get("VL-Time-Field"); tf != "" {

						timeField = tf

					}

					// Extract message field name from _msg_field query arg or header

					msgField := ""

					// Extract message field name from _msg_field query arg

					var msgField = ""

					if msgf := r.FormValue("_msg_field"); msgf != "" {

						msgField = msgf

					} else if msgf = r.Header.Get("VL-Msg-Field"); msgf != "" {

						msgField = msgf

					}

					streamFields := httputils.GetArray(r, "_stream_fields")

					if len(streamFields) == 0 {

						if sf := r.Header.Get("VL-Stream-Fields"); len(sf) > 0 {

							streamFields = strings.Split(sf, ",")

						}

					}

					ignoreFields := httputils.GetArray(r, "ignore_fields")

					if len(ignoreFields) == 0 {

						if f := r.Header.Get("VL-Ignore-Fields"); len(f) > 0 {

							ignoreFields = strings.Split(f, ",")

						}

					}

					debug := httputils.GetBool(r, "debug")

					if !debug {

						if dh := r.Header.Get("VL-Debug"); len(dh) > 0 {

							hv := strings.ToLower(dh)

							switch hv {

							case "", "0", "f", "false", "no":

							default:

								debug = true

							}

						}

					}

					debugRequestURI := ""

					debugRemoteAddr := ""

					if debug {

				@@ -96,7 +71,6 @@ func GetCommonParams(r *http.Request) (*CommonParams, error) {

						DebugRequestURI: debugRequestURI,

						DebugRemoteAddr: debugRemoteAddr,

					}

					return cp, nil

				}

				@@ -176,27 +150,11 @@ func (lmp *logMessageProcessor) AddRow(timestamp int64, fields []logstorage.Fiel

						return

					}

					// _msg field must be non-empty according to VictoriaLogs data model.

					// See https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field

					msgExist := false

					for i := range fields {

						if fields[i].Name == "_msg" {

							msgExist = len(fields[i].Value) > 0

							break

						}

					}

					if !msgExist {

						rf := logstorage.RowFormatter(fields)

						logger.Warnf("dropping log line without _msg field; %s", rf)

						rowsDroppedTotalMsgNotValid.Inc()

						return

					}

					lmp.lr.MustAdd(lmp.cp.TenantID, timestamp, fields)

					if lmp.cp.Debug {

						s := lmp.lr.GetRowString(0)

						lmp.lr.ResetKeepSettings()

						logger.Infof("remoteAddr=%s; requestURI=%s; ignoring log entry because of `debug` arg: %s", lmp.cp.DebugRemoteAddr, lmp.cp.DebugRequestURI, s)

						logger.Infof("remoteAddr=%s; requestURI=%s; ignoring log entry because of `debug` query arg: %s", lmp.cp.DebugRemoteAddr, lmp.cp.DebugRequestURI, s)

						rowsDroppedTotalDebug.Inc()

						return

					}

				@@ -238,8 +196,5 @@ func (cp *CommonParams) NewLogMessageProcessor() LogMessageProcessor {

					return lmp

				}

				var (

					rowsDroppedTotalDebug         = metrics.NewCounter(`vl_rows_dropped_total{reason="debug"}`)

					rowsDroppedTotalTooManyFields = metrics.NewCounter(`vl_rows_dropped_total{reason="too_many_fields"}`)

					rowsDroppedTotalMsgNotValid   = metrics.NewCounter(`vl_rows_dropped_total{reason="msg_not_exist"}`)

				)

				var rowsDroppedTotalDebug = metrics.NewCounter(`vl_rows_dropped_total{reason="debug"}`)

				var rowsDroppedTotalTooManyFields = metrics.NewCounter(`vl_rows_dropped_total{reason="too_many_fields"}`)

									
										48

app/vlinsert/insertutils/timestamp.go
									
												View File
												
				@@ -2,8 +2,6 @@ package insertutils

				import (

					"fmt"

					"math"

					"strconv"

					"time"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"

				@@ -21,49 +19,15 @@ func ExtractTimestampRFC3339NanoFromFields(timeField string, fields []logstorage

						if f.Name != timeField {

							continue

						}

						nsecs, err := parseTimestamp(f.Value)

						if err != nil {

							return 0, fmt.Errorf("cannot parse timestamp from field %q: %s", timeField, err)

						if f.Value == "" || f.Value == "0" {

							return time.Now().UnixNano(), nil

						}

						nsecs, ok := logstorage.TryParseTimestampRFC3339Nano(f.Value)

						if !ok {

							return 0, fmt.Errorf("cannot unmarshal rfc3339 timestamp from %s=%q", timeField, f.Value)

						}

						f.Value = ""

						if nsecs == 0 {

							nsecs = time.Now().UnixNano()

						}

						return nsecs, nil

					}

					return time.Now().UnixNano(), nil

				}

				func parseTimestamp(s string) (int64, error) {

					if s == "" || s == "0" {

						return time.Now().UnixNano(), nil

					}

					if len(s) <= len("YYYY") || s[len("YYYY")] != '-' {

						return ParseUnixTimestamp(s)

					}

					nsecs, ok := logstorage.TryParseTimestampRFC3339Nano(s)

					if !ok {

						return 0, fmt.Errorf("cannot unmarshal rfc3339 timestamp %q", s)

					}

					return nsecs, nil

				}

				// ParseUnixTimestamp parses s as unix timestamp in either seconds or milliseconds and returns the parsed timestamp in nanoseconds.

				func ParseUnixTimestamp(s string) (int64, error) {

					n, err := strconv.ParseInt(s, 10, 64)

					if err != nil {

						return 0, fmt.Errorf("cannot parse unix timestamp from %q: %w", s, err)

					}

					if n < (1<<31) && n >= (-1<<31) {

						// The timestamp is in seconds. Convert it to milliseconds

						n *= 1e3

					}

					if n > int64(math.MaxInt64)/1e6 {

						return 0, fmt.Errorf("too big timestamp in milliseconds: %d; mustn't exceed %d", n, int64(math.MaxInt64)/1e6)

					}

					if n < int64(math.MinInt64)/1e6 {

						return 0, fmt.Errorf("too small timestamp in milliseconds: %d; must be bigger than %d", n, int64(math.MinInt64)/1e6)

					}

					n *= 1e6

					return n, nil

				}

									
										21

app/vlinsert/insertutils/timestamp_test.go
									
												View File
												
				@@ -27,41 +27,25 @@ func TestExtractTimestampRFC3339NanoFromFields_Success(t *testing.T) {

						}

					}

					// UTC time

					f("time", []logstorage.Field{

						{Name: "foo", Value: "bar"},

						{Name: "time", Value: "2024-06-18T23:37:20Z"},

					}, 1718753840000000000)

					// Time with timezone

					f("time", []logstorage.Field{

						{Name: "foo", Value: "bar"},

						{Name: "time", Value: "2024-06-18T23:37:20+08:00"},

					}, 1718725040000000000)

					// SQL datetime format

					f("time", []logstorage.Field{

						{Name: "foo", Value: "bar"},

						{Name: "time", Value: "2024-06-18 23:37:20.123-05:30"},

						{Name: "time", Value: "2024-06-18T23:37:20.123-05:30"},

					}, 1718773640123000000)

					// Time with nanosecond precision

					f("time", []logstorage.Field{

						{Name: "time", Value: "2024-06-18T23:37:20.123456789-05:30"},

						{Name: "foo", Value: "bar"},

					}, 1718773640123456789)

					// Unix timestamp in milliseconds

					f("time", []logstorage.Field{

						{Name: "foo", Value: "bar"},

						{Name: "time", Value: "1718773640123"},

					}, 1718773640123000000)

					// Unix timestamp in seconds

					f("time", []logstorage.Field{

						{Name: "foo", Value: "bar"},

						{Name: "time", Value: "1718773640"},

					}, 1718773640000000000)

				}

				func TestExtractTimestampRFC3339NanoFromFields_Error(t *testing.T) {

				@@ -82,6 +66,9 @@ func TestExtractTimestampRFC3339NanoFromFields_Error(t *testing.T) {

					f("foobar")

					// no Z at the end

					f("2024-06-18T23:37:20")

					// incomplete time

					f("2024-06-18")

					f("2024-06-18T23:37")

									
										12

app/vlinsert/jsonline/jsonline_test.go
									
												View File
												
				@@ -23,16 +23,16 @@ func TestProcessStreamInternal_Success(t *testing.T) {

					}

					data := `{"@timestamp":"2023-06-06T04:48:11.735Z","log":{"offset":71770,"file":{"path":"/var/log/auth.log"}},"message":"foobar"}

				{"@timestamp":"2023-06-06T04:48:12.735+01:00","message":"baz"}

				{"message":"xyz","@timestamp":"2023-06-06 04:48:13.735Z","x":"y"}

				{"@timestamp":"2023-06-06T04:48:12.735Z","message":"baz"}

				{"message":"xyz","@timestamp":"2023-06-06T04:48:13.735Z","x":"y"}

				`

					timeField := "@timestamp"

					msgField := "message"

					rowsExpected := 3

					timestampsExpected := []int64{1686026891735000000, 1686023292735000000, 1686026893735000000}

					resultExpected := `{"log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}

				{"_msg":"baz"}

				{"_msg":"xyz","x":"y"}`

					timestampsExpected := []int64{1686026891735000000, 1686026892735000000, 1686026893735000000}

					resultExpected := `{"@timestamp":"","log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}

				{"@timestamp":"","_msg":"baz"}

				{"_msg":"xyz","@timestamp":"","x":"y"}`

					f(data, timeField, msgField, rowsExpected, timestampsExpected, resultExpected)

				}

									
										4

app/vlinsert/main.go
									
												View File
												
				@@ -7,7 +7,6 @@ import (

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/elasticsearch"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/jsonline"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/loki"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/opentelemetry"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/syslog"

				)

				@@ -42,9 +41,6 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {

					case strings.HasPrefix(path, "/loki/"):

						path = strings.TrimPrefix(path, "/loki")

						return loki.RequestHandler(path, w, r)

					case strings.HasPrefix(path, "/opentelemetry/"):

						path = strings.TrimPrefix(path, "/opentelemetry")

						return opentelemetry.RequestHandler(path, w, r)

					default:

						return false

					}

									
										143

app/vlinsert/opentelemetry/opentelemetry.go
									
												View File
											
				@@ -1,143 +0,0 @@

				package opentelemetry

				import (

					"fmt"

					"io"

					"net/http"

					"time"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/pb"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/slicesutil"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"

					"github.com/VictoriaMetrics/metrics"

				)

				// RequestHandler processes Opentelemetry insert requests

				func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {

					switch path {

					// use the same path as opentelemetry collector

					// https://opentelemetry.io/docs/specs/otlp/#otlphttp-request

					case "/v1/logs":

						if r.Header.Get("Content-Type") == "application/json" {

							httpserver.Errorf(w, r, "json encoding isn't supported for opentelemetry format. Use protobuf encoding")

							return true

						}

						handleProtobuf(r, w)

						return true

					default:

						return false

					}

				}

				func handleProtobuf(r *http.Request, w http.ResponseWriter) {

					startTime := time.Now()

					requestsProtobufTotal.Inc()

					reader := r.Body

					if r.Header.Get("Content-Encoding") == "gzip" {

						zr, err := common.GetGzipReader(reader)

						if err != nil {

							httpserver.Errorf(w, r, "cannot initialize gzip reader: %s", err)

							return

						}

						defer common.PutGzipReader(zr)

						reader = zr

					}

					wcr := writeconcurrencylimiter.GetReader(reader)

					data, err := io.ReadAll(wcr)

					writeconcurrencylimiter.PutReader(wcr)

					if err != nil {

						httpserver.Errorf(w, r, "cannot read request body: %s", err)

						return

					}

					cp, err := insertutils.GetCommonParams(r)

					if err != nil {

						httpserver.Errorf(w, r, "cannot parse common params from request: %s", err)

						return

					}

					if err := vlstorage.CanWriteData(); err != nil {

						httpserver.Errorf(w, r, "%s", err)

						return

					}

					lmp := cp.NewLogMessageProcessor()

					n, err := pushProtobufRequest(data, lmp)

					lmp.MustClose()

					if err != nil {

						httpserver.Errorf(w, r, "cannot parse OpenTelemetry protobuf request: %s", err)

						return

					}

					rowsIngestedProtobufTotal.Add(n)

					// update requestProtobufDuration only for successfully parsed requests

					// There is no need in updating requestProtobufDuration for request errors,

					// since their timings are usually much smaller than the timing for successful request parsing.

					requestProtobufDuration.UpdateDuration(startTime)

				}

				var (

					rowsIngestedProtobufTotal = metrics.NewCounter(`vl_rows_ingested_total{type="opentelemetry",format="protobuf"}`)

					requestsProtobufTotal = metrics.NewCounter(`vl_http_requests_total{path="/insert/opentelemetry/v1/logs",format="protobuf"}`)

					errorsTotal           = metrics.NewCounter(`vl_http_errors_total{path="/insert/opentelemetry/v1/logs",format="protobuf"}`)

					requestProtobufDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/opentelemetry/v1/logs",format="protobuf"}`)

				)

				func pushProtobufRequest(data []byte, lmp insertutils.LogMessageProcessor) (int, error) {

					var req pb.ExportLogsServiceRequest

					if err := req.UnmarshalProtobuf(data); err != nil {

						errorsTotal.Inc()

						return 0, fmt.Errorf("cannot unmarshal request from %d bytes: %w", len(data), err)

					}

					var rowsIngested int

					var commonFields []logstorage.Field

					for _, rl := range req.ResourceLogs {

						attributes := rl.Resource.Attributes

						commonFields = slicesutil.SetLength(commonFields, len(attributes))

						for i, attr := range attributes {

							commonFields[i].Name = attr.Key

							commonFields[i].Value = attr.Value.FormatString()

						}

						commonFieldsLen := len(commonFields)

						for _, sc := range rl.ScopeLogs {

							var scopeIngested int

							commonFields, scopeIngested = pushFieldsFromScopeLogs(&sc, commonFields[:commonFieldsLen], lmp)

							rowsIngested += scopeIngested

						}

					}

					return rowsIngested, nil

				}

				func pushFieldsFromScopeLogs(sc *pb.ScopeLogs, commonFields []logstorage.Field, lmp insertutils.LogMessageProcessor) ([]logstorage.Field, int) {

					fields := commonFields

					for _, lr := range sc.LogRecords {

						fields = fields[:len(commonFields)]

						fields = append(fields, logstorage.Field{

							Name:  "_msg",

							Value: lr.Body.FormatString(),

						})

						for _, attr := range lr.Attributes {

							fields = append(fields, logstorage.Field{

								Name:  attr.Key,

								Value: attr.Value.FormatString(),

							})

						}

						fields = append(fields, logstorage.Field{

							Name:  "severity",

							Value: lr.FormatSeverity(),

						})

						lmp.AddRow(lr.ExtractTimestampNano(), fields)

					}

					return fields, len(sc.LogRecords)

				}

									
										128

app/vlinsert/opentelemetry/opentelemetry_test.go
									
												View File
											
				@@ -1,128 +0,0 @@

				package opentelemetry

				import (

					"testing"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/pb"

				)

				func TestPushProtoOk(t *testing.T) {

					f := func(src []pb.ResourceLogs, timestampsExpected []int64, resultExpected string) {

						t.Helper()

						lr := pb.ExportLogsServiceRequest{

							ResourceLogs: src,

						}

						pData := lr.MarshalProtobuf(nil)

						tlp := &insertutils.TestLogMessageProcessor{}

						n, err := pushProtobufRequest(pData, tlp)

						if err != nil {

							t.Fatalf("unexpected error: %s", err)

						}

						if err := tlp.Verify(n, timestampsExpected, resultExpected); err != nil {

							t.Fatal(err)

						}

					}

					// single line without resource attributes

					f([]pb.ResourceLogs{

						{

							ScopeLogs: []pb.ScopeLogs{

								{

									LogRecords: []pb.LogRecord{

										{Attributes: []*pb.KeyValue{}, TimeUnixNano: 1234, SeverityNumber: 1, Body: pb.AnyValue{StringValue: ptrTo("log-line-message")}},

									},

								},

							},

						},

					},

						[]int64{1234},

						`{"_msg":"log-line-message","severity":"Trace"}`,

					)

					// multi-line with resource attributes

					f([]pb.ResourceLogs{

						{

							Resource: pb.Resource{

								Attributes: []*pb.KeyValue{

									{Key: "logger", Value: &pb.AnyValue{StringValue: ptrTo("context")}},

									{Key: "instance_id", Value: &pb.AnyValue{IntValue: ptrTo[int64](10)}},

									{Key: "node_taints", Value: &pb.AnyValue{KeyValueList: &pb.KeyValueList{

										Values: []*pb.KeyValue{

											{Key: "role", Value: &pb.AnyValue{StringValue: ptrTo("dev")}},

											{Key: "cluster_load_percent", Value: &pb.AnyValue{DoubleValue: ptrTo(0.55)}},

										},

									}}},

								},

							},

							ScopeLogs: []pb.ScopeLogs{

								{

									LogRecords: []pb.LogRecord{

										{Attributes: []*pb.KeyValue{}, TimeUnixNano: 1234, SeverityNumber: 1, Body: pb.AnyValue{StringValue: ptrTo("log-line-message")}},

										{Attributes: []*pb.KeyValue{}, TimeUnixNano: 1235, SeverityNumber: 21, Body: pb.AnyValue{StringValue: ptrTo("log-line-message-msg-2")}},

										{Attributes: []*pb.KeyValue{}, TimeUnixNano: 1236, SeverityNumber: -1, Body: pb.AnyValue{StringValue: ptrTo("log-line-message-msg-2")}},

									},

								},

							},

						},

					},

						[]int64{1234, 1235, 1236},

						`{"logger":"context","instance_id":"10","node_taints":"[{\"Key\":\"role\",\"Value\":{\"StringValue\":\"dev\",\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":null,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}},{\"Key\":\"cluster_load_percent\",\"Value\":{\"StringValue\":null,\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":0.55,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}}]","_msg":"log-line-message","severity":"Trace"}

				{"logger":"context","instance_id":"10","node_taints":"[{\"Key\":\"role\",\"Value\":{\"StringValue\":\"dev\",\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":null,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}},{\"Key\":\"cluster_load_percent\",\"Value\":{\"StringValue\":null,\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":0.55,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}}]","_msg":"log-line-message-msg-2","severity":"Unspecified"}

				{"logger":"context","instance_id":"10","node_taints":"[{\"Key\":\"role\",\"Value\":{\"StringValue\":\"dev\",\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":null,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}},{\"Key\":\"cluster_load_percent\",\"Value\":{\"StringValue\":null,\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":0.55,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}}]","_msg":"log-line-message-msg-2","severity":"Unspecified"}`,

					)

					// multi-scope with resource attributes and multi-line

					f([]pb.ResourceLogs{

						{

							Resource: pb.Resource{

								Attributes: []*pb.KeyValue{

									{Key: "logger", Value: &pb.AnyValue{StringValue: ptrTo("context")}},

									{Key: "instance_id", Value: &pb.AnyValue{IntValue: ptrTo[int64](10)}},

									{Key: "node_taints", Value: &pb.AnyValue{KeyValueList: &pb.KeyValueList{

										Values: []*pb.KeyValue{

											{Key: "role", Value: &pb.AnyValue{StringValue: ptrTo("dev")}},

											{Key: "cluster_load_percent", Value: &pb.AnyValue{DoubleValue: ptrTo(0.55)}},

										},

									}}},

								},

							},

							ScopeLogs: []pb.ScopeLogs{

								{

									LogRecords: []pb.LogRecord{

										{TimeUnixNano: 1234, SeverityNumber: 1, Body: pb.AnyValue{StringValue: ptrTo("log-line-message")}},

										{TimeUnixNano: 1235, SeverityNumber: 5, Body: pb.AnyValue{StringValue: ptrTo("log-line-message-msg-2")}},

									},

								},

							},

						},

						{

							ScopeLogs: []pb.ScopeLogs{

								{

									LogRecords: []pb.LogRecord{

										{TimeUnixNano: 2345, SeverityNumber: 10, Body: pb.AnyValue{StringValue: ptrTo("log-line-resource-scope-1-0-0")}},

										{TimeUnixNano: 2346, SeverityNumber: 10, Body: pb.AnyValue{StringValue: ptrTo("log-line-resource-scope-1-0-1")}},

									},

								},

								{

									LogRecords: []pb.LogRecord{

										{TimeUnixNano: 2347, SeverityNumber: 12, Body: pb.AnyValue{StringValue: ptrTo("log-line-resource-scope-1-1-0")}},

										{ObservedTimeUnixNano: 2348, SeverityNumber: 12, Body: pb.AnyValue{StringValue: ptrTo("log-line-resource-scope-1-1-1")}},

									},

								},

							},

						},

					},

						[]int64{1234, 1235, 2345, 2346, 2347, 2348},

						`{"logger":"context","instance_id":"10","node_taints":"[{\"Key\":\"role\",\"Value\":{\"StringValue\":\"dev\",\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":null,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}},{\"Key\":\"cluster_load_percent\",\"Value\":{\"StringValue\":null,\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":0.55,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}}]","_msg":"log-line-message","severity":"Trace"}

				{"logger":"context","instance_id":"10","node_taints":"[{\"Key\":\"role\",\"Value\":{\"StringValue\":\"dev\",\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":null,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}},{\"Key\":\"cluster_load_percent\",\"Value\":{\"StringValue\":null,\"BoolValue\":null,\"IntValue\":null,\"DoubleValue\":0.55,\"ArrayValue\":null,\"KeyValueList\":null,\"BytesValue\":null}}]","_msg":"log-line-message-msg-2","severity":"Debug"}

				{"_msg":"log-line-resource-scope-1-0-0","severity":"Info2"}

				{"_msg":"log-line-resource-scope-1-0-1","severity":"Info2"}

				{"_msg":"log-line-resource-scope-1-1-0","severity":"Info4"}

				{"_msg":"log-line-resource-scope-1-1-1","severity":"Info4"}`,

					)

				}

				func ptrTo[T any](s T) *T {

					return &s

				}

									
										79

app/vlinsert/opentelemetry/opentemetry_timing_test.go
									
												View File
											
				@@ -1,79 +0,0 @@

				package opentelemetry

				import (

					"fmt"

					"testing"

					"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/opentelemetry/pb"

				)

				func BenchmarkParseProtobufRequest(b *testing.B) {

					for _, scopes := range []int{1, 2} {

						for _, rows := range []int{100, 1000} {

							for _, attributes := range []int{5, 10} {

								b.Run(fmt.Sprintf("scopes_%d/rows_%d/attributes_%d", scopes, rows, attributes), func(b *testing.B) {

									benchmarkParseProtobufRequest(b, scopes, rows, attributes)

								})

							}

						}

					}

				}

				func benchmarkParseProtobufRequest(b *testing.B, streams, rows, labels int) {

					blp := &insertutils.BenchmarkLogMessageProcessor{}

					b.ReportAllocs()

					b.SetBytes(int64(streams * rows))

					b.RunParallel(func(pb *testing.PB) {

						body := getProtobufBody(streams, rows, labels)

						for pb.Next() {

							_, err := pushProtobufRequest(body, blp)

							if err != nil {

								panic(fmt.Errorf("unexpected error: %w", err))

							}

						}

					})

				}

				func getProtobufBody(scopesCount, rowsCount, attributesCount int) []byte {

					msg := "12345678910"

					attrValues := []*pb.AnyValue{

						{StringValue: ptrTo("string-attribute")},

						{IntValue: ptrTo[int64](12345)},

						{DoubleValue: ptrTo(3.14)},

					}

					attrs := make([]*pb.KeyValue, attributesCount)

					for j := 0; j < attributesCount; j++ {

						attrs[j] = &pb.KeyValue{

							Key:   fmt.Sprintf("key-%d", j),

							Value: attrValues[j%3],

						}

					}

					entries := make([]pb.LogRecord, rowsCount)

					for j := 0; j < rowsCount; j++ {

						entries[j] = pb.LogRecord{

							TimeUnixNano: 12345678910, ObservedTimeUnixNano: 12345678910, Body: pb.AnyValue{StringValue: &msg},

						}

					}

					scopes := make([]pb.ScopeLogs, scopesCount)

					for j := 0; j < scopesCount; j++ {

						scopes[j] = pb.ScopeLogs{

							LogRecords: entries,

						}

					}

					pr := pb.ExportLogsServiceRequest{

						ResourceLogs: []pb.ResourceLogs{

							{

								Resource: pb.Resource{

									Attributes: attrs,

								},

								ScopeLogs: scopes,

							},

						},

					}

					return pr.MarshalProtobuf(nil)

				}

									
										6

app/vlinsert/syslog/syslog_test.go
									
												View File
												
				@@ -101,9 +101,9 @@ func TestProcessStreamInternal_Success(t *testing.T) {

					currentYear := 2023

					rowsExpected := 3

					timestampsExpected := []int64{1685794113000000000, 1685880513000000000, 1685814132345000000}

					resultExpected := `{"format":"rfc3164","hostname":"abcd","app_name":"systemd","_msg":"Starting Update the local ESM caches..."}

				{"priority":"165","facility":"20","severity":"5","format":"rfc3164","hostname":"abcd","app_name":"systemd","proc_id":"345","_msg":"abc defg"}

				{"priority":"123","facility":"15","severity":"3","format":"rfc5424","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`

					resultExpected := `{"format":"rfc3164","timestamp":"","hostname":"abcd","app_name":"systemd","_msg":"Starting Update the local ESM caches..."}

				{"priority":"165","facility":"20","severity":"5","format":"rfc3164","timestamp":"","hostname":"abcd","app_name":"systemd","proc_id":"345","_msg":"abc defg"}

				{"priority":"123","facility":"15","severity":"3","format":"rfc5424","timestamp":"","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`

					f(data, currentYear, rowsExpected, timestampsExpected, resultExpected)

				}

									
										109

app/vlogscli/Makefile
									
												View File
											
				@@ -1,109 +0,0 @@

				# All these commands must run from repository root.

				vlogscli:

					APP_NAME=vlogscli $(MAKE) app-local

				vlogscli-race:

					APP_NAME=vlogscli RACE=-race $(MAKE) app-local

				vlogscli-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker

				vlogscli-pure-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-pure

				vlogscli-linux-amd64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-linux-amd64

				vlogscli-linux-arm-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-linux-arm

				vlogscli-linux-arm64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-linux-arm64

				vlogscli-linux-ppc64le-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-linux-ppc64le

				vlogscli-linux-386-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-linux-386

				vlogscli-darwin-amd64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-darwin-amd64

				vlogscli-darwin-arm64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-darwin-arm64

				vlogscli-freebsd-amd64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-freebsd-amd64

				vlogscli-openbsd-amd64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-openbsd-amd64

				vlogscli-windows-amd64-prod:

					APP_NAME=vlogscli $(MAKE) app-via-docker-windows-amd64

				package-vlogscli:

					APP_NAME=vlogscli $(MAKE) package-via-docker

				package-vlogscli-pure:

					APP_NAME=vlogscli $(MAKE) package-via-docker-pure

				package-vlogscli-amd64:

					APP_NAME=vlogscli $(MAKE) package-via-docker-amd64

				package-vlogscli-arm:

					APP_NAME=vlogscli $(MAKE) package-via-docker-arm

				package-vlogscli-arm64:

					APP_NAME=vlogscli $(MAKE) package-via-docker-arm64

				package-vlogscli-ppc64le:

					APP_NAME=vlogscli $(MAKE) package-via-docker-ppc64le

				package-vlogscli-386:

					APP_NAME=vlogscli $(MAKE) package-via-docker-386

				publish-vlogscli:

					APP_NAME=vlogscli $(MAKE) publish-via-docker

				vlogscli-linux-amd64:

					APP_NAME=vlogscli CGO_ENABLED=1 GOOS=linux GOARCH=amd64 $(MAKE) app-local-goos-goarch

				vlogscli-linux-arm:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=arm $(MAKE) app-local-goos-goarch

				vlogscli-linux-arm64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=arm64 $(MAKE) app-local-goos-goarch

				vlogscli-linux-ppc64le:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le $(MAKE) app-local-goos-goarch

				vlogscli-linux-s390x:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch

				vlogscli-linux-loong64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch

				vlogscli-linux-386:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch

				vlogscli-darwin-amd64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 $(MAKE) app-local-goos-goarch

				vlogscli-darwin-arm64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 $(MAKE) app-local-goos-goarch

				vlogscli-freebsd-amd64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 $(MAKE) app-local-goos-goarch

				vlogscli-openbsd-amd64:

					APP_NAME=vlogscli CGO_ENABLED=0 GOOS=openbsd GOARCH=amd64 $(MAKE) app-local-goos-goarch

				vlogscli-windows-amd64:

					GOARCH=amd64 APP_NAME=vlogscli $(MAKE) app-local-windows-goarch

				vlogscli-pure:

					APP_NAME=vlogscli $(MAKE) app-local-pure

				run-vlogscli:

					APP_NAME=vlogscli $(MAKE) run-via-docker

									
										5

app/vlogscli/README.md
									
												View File
											
				@@ -1,5 +0,0 @@

				# vlogscli

				Command-line utility for querying [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/).

				See [these docs](https://docs.victoriametrics.com/victorialogs/querying/vlogscli/).

									
										6

app/vlogscli/deployment/Dockerfile
									
												View File
											
				@@ -1,6 +0,0 @@

				ARG base_image=non-existing

				FROM $base_image

				ENTRYPOINT ["/vlogscli-prod"]

				ARG src_binary=non-existing

				COPY $src_binary ./vlogscli-prod

									
										238

app/vlogscli/json_prettifier.go
									
												View File
											
				@@ -1,238 +0,0 @@

				package main

				import (

					"bufio"

					"encoding/json"

					"fmt"

					"io"

					"sort"

					"sync"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"

				)

				type outputMode int

				const (

					outputModeJSONMultiline  = outputMode(0)

					outputModeJSONSingleline = outputMode(1)

					outputModeLogfmt         = outputMode(2)

					outputModeCompact        = outputMode(3)

				)

				func getOutputFormatter(outputMode outputMode) func(w io.Writer, fields []logstorage.Field) error {

					switch outputMode {

					case outputModeJSONMultiline:

						return func(w io.Writer, fields []logstorage.Field) error {

							return writeJSONObject(w, fields, true)

						}

					case outputModeJSONSingleline:

						return func(w io.Writer, fields []logstorage.Field) error {

							return writeJSONObject(w, fields, false)

						}

					case outputModeLogfmt:

						return writeLogfmtObject

					case outputModeCompact:

						return writeCompactObject

					default:

						panic(fmt.Errorf("BUG: unexpected outputMode=%d", outputMode))

					}

				}

				type jsonPrettifier struct {

					r         io.ReadCloser

					formatter func(w io.Writer, fields []logstorage.Field) error

					d *json.Decoder

					pr *io.PipeReader

					pw *io.PipeWriter

					bw *bufio.Writer

					wg sync.WaitGroup

				}

				func newJSONPrettifier(r io.ReadCloser, outputMode outputMode) *jsonPrettifier {

					d := json.NewDecoder(r)

					pr, pw := io.Pipe()

					bw := bufio.NewWriter(pw)

					formatter := getOutputFormatter(outputMode)

					jp := &jsonPrettifier{

						r:         r,

						formatter: formatter,

						d: d,

						pr: pr,

						pw: pw,

						bw: bw,

					}

					jp.wg.Add(1)

					go func() {

						defer jp.wg.Done()

						err := jp.prettifyJSONLines()

						jp.closePipesWithError(err)

					}()

					return jp

				}

				func (jp *jsonPrettifier) closePipesWithError(err error) {

					_ = jp.pr.CloseWithError(err)

					_ = jp.pw.CloseWithError(err)

				}

				func (jp *jsonPrettifier) prettifyJSONLines() error {

					for jp.d.More() {

						fields, err := readNextJSONObject(jp.d)

						if err != nil {

							return err

						}

						sort.Slice(fields, func(i, j int) bool {

							return fields[i].Name < fields[j].Name

						})

						if err := jp.formatter(jp.bw, fields); err != nil {

							return err

						}

						// Flush bw after every output line in order to show results as soon as they appear.

						if err := jp.bw.Flush(); err != nil {

							return err

						}

					}

					return nil

				}

				func (jp *jsonPrettifier) Close() error {

					jp.closePipesWithError(io.ErrUnexpectedEOF)

					err := jp.r.Close()

					jp.wg.Wait()

					return err

				}

				func (jp *jsonPrettifier) Read(p []byte) (int, error) {

					return jp.pr.Read(p)

				}

				func readNextJSONObject(d *json.Decoder) ([]logstorage.Field, error) {

					t, err := d.Token()

					if err != nil {

						return nil, fmt.Errorf("cannot read '{': %w", err)

					}

					delim, ok := t.(json.Delim)

					if !ok || delim.String() != "{" {

						return nil, fmt.Errorf("unexpected token read; got %q; want '{'", delim)

					}

					var fields []logstorage.Field

					for {

						// Read object key

						t, err := d.Token()

						if err != nil {

							return nil, fmt.Errorf("cannot read JSON object key or closing brace: %w", err)

						}

						delim, ok := t.(json.Delim)

						if ok {

							if delim.String() == "}" {

								return fields, nil

							}

							return nil, fmt.Errorf("unexpected delimiter read; got %q; want '}'", delim)

						}

						key, ok := t.(string)

						if !ok {

							return nil, fmt.Errorf("unexpected token read for object key: %v; want string or '}'", t)

						}

						// read object value

						t, err = d.Token()

						if err != nil {

							return nil, fmt.Errorf("cannot read JSON object value: %w", err)

						}

						value, ok := t.(string)

						if !ok {

							return nil, fmt.Errorf("unexpected token read for oject value: %v; want string", t)

						}

						fields = append(fields, logstorage.Field{

							Name:  key,

							Value: value,

						})

					}

				}

				func writeLogfmtObject(w io.Writer, fields []logstorage.Field) error {

					data := logstorage.MarshalFieldsToLogfmt(nil, fields)

					_, err := fmt.Fprintf(w, "%s\n", data)

					return err

				}

				func writeCompactObject(w io.Writer, fields []logstorage.Field) error {

					if len(fields) == 1 {

						// Just write field value as is without name

						_, err := fmt.Fprintf(w, "%s\n", fields[0].Value)

						return err

					}

					if len(fields) == 2 && fields[0].Name == "_time" || fields[1].Name == "_time" {

						// Write _time\tfieldValue as is

						if fields[0].Name == "_time" {

							_, err := fmt.Fprintf(w, "%s\t%s\n", fields[0].Value, fields[1].Value)

							return err

						}

						_, err := fmt.Fprintf(w, "%s\t%s\n", fields[1].Value, fields[0].Value)

						return err

					}

					// Fall back to logfmt

					return writeLogfmtObject(w, fields)

				}

				func writeJSONObject(w io.Writer, fields []logstorage.Field, isMultiline bool) error {

					if len(fields) == 0 {

						fmt.Fprintf(w, "{}\n")

						return nil

					}

					fmt.Fprintf(w, "{")

					writeNewlineIfNeeded(w, isMultiline)

					if err := writeJSONObjectKeyValue(w, fields[0], isMultiline); err != nil {

						return err

					}

					for _, f := range fields[1:] {

						fmt.Fprintf(w, ",")

						writeNewlineIfNeeded(w, isMultiline)

						if err := writeJSONObjectKeyValue(w, f, isMultiline); err != nil {

							return err

						}

					}

					writeNewlineIfNeeded(w, isMultiline)

					fmt.Fprintf(w, "}\n")

					return nil

				}

				func writeNewlineIfNeeded(w io.Writer, isMultiline bool) {

					if isMultiline {

						fmt.Fprintf(w, "\n")

					}

				}

				func writeJSONObjectKeyValue(w io.Writer, f logstorage.Field, isMultiline bool) error {

					key := getJSONString(f.Name)

					value := getJSONString(f.Value)

					if isMultiline {

						_, err := fmt.Fprintf(w, "  %s: %s", key, value)

						return err

					}

					_, err := fmt.Fprintf(w, "%s:%s", key, value)

					return err

				}

				func getJSONString(s string) string {

					data, err := json.Marshal(s)

					if err != nil {

						panic(fmt.Errorf("unexpected error when marshaling string to JSON: %w", err))

					}

					return string(data)

				}

									
										116

app/vlogscli/less_wrapper.go
									
												View File
											
				@@ -1,116 +0,0 @@

				package main

				import (

					"errors"

					"fmt"

					"io"

					"os"

					"os/exec"

					"os/signal"

					"sync"

					"syscall"

					"github.com/mattn/go-isatty"

				)

				func isTerminal() bool {

					return isatty.IsTerminal(os.Stdout.Fd()) && isatty.IsTerminal(os.Stderr.Fd())

				}

				func readWithLess(r io.Reader) error {

					if !isTerminal() {

						// Just write everything to stdout if no terminal is available.

						_, err := io.Copy(os.Stdout, r)

						if err != nil && !isErrPipe(err) {

							return fmt.Errorf("error when forwarding data to stdout: %w", err)

						}

						if err := os.Stdout.Sync(); err != nil {

							return fmt.Errorf("cannot sync data to stdout: %w", err)

						}

						return nil

					}

					pr, pw, err := os.Pipe()

					if err != nil {

						return fmt.Errorf("cannot create pipe: %w", err)

					}

					defer func() {

						_ = pr.Close()

						_ = pw.Close()

					}()

					// Ignore Ctrl+C in the current process, so 'less' could handle it properly

					cancel := ignoreSignals(os.Interrupt)

					defer cancel()

					// Start 'less' process

					path, err := exec.LookPath("less")

					if err != nil {

						return fmt.Errorf("cannot find 'less' command: %w", err)

					}

					p, err := os.StartProcess(path, []string{"less", "-F", "-X"}, &os.ProcAttr{

						Env:   append(os.Environ(), "LESSCHARSET=utf-8"),

						Files: []*os.File{pr, os.Stdout, os.Stderr},

					})

					if err != nil {

						return fmt.Errorf("cannot start 'less' process: %w", err)

					}

					// Close pr after 'less' finishes in a parallel goroutine

					// in order to unblock forwarding data to stopped 'less' below.

					waitch := make(chan *os.ProcessState)

					go func() {

						// Wait for 'less' process to finish.

						ps, err := p.Wait()

						if err != nil {

							fatalf("unexpected error when waiting for 'less' process: %w", err)

						}

						_ = pr.Close()

						waitch <- ps

					}()

					// Forward data from r to 'less'

					_, err = io.Copy(pw, r)

					_ = pw.Sync()

					_ = pw.Close()

					// Wait until 'less' finished

					ps := <-waitch

					// Verify 'less' status.

					if !ps.Success() {

						return fmt.Errorf("'less' finished with unexpected code %d", ps.ExitCode())

					}

					if err != nil && !isErrPipe(err) {

						return fmt.Errorf("error when forwarding data to 'less': %w", err)

					}

					return nil

				}

				func isErrPipe(err error) bool {

					return errors.Is(err, syscall.EPIPE) || errors.Is(err, io.ErrClosedPipe)

				}

				func ignoreSignals(sigs ...os.Signal) func() {

					ch := make(chan os.Signal, 1)

					signal.Notify(ch, sigs...)

					var wg sync.WaitGroup

					wg.Add(1)

					go func() {

						defer wg.Done()

						for {

							_, ok := <-ch

							if !ok {

								return

							}

						}

					}()

					return func() {

						signal.Stop(ch)

						close(ch)

						wg.Wait()

					}

				}

									
										420

app/vlogscli/main.go
									
												View File
											
				@@ -1,420 +0,0 @@

				package main

				import (

					"context"

					"errors"

					"flag"

					"fmt"

					"io"

					"io/fs"

					"net/http"

					"net/url"

					"os"

					"os/signal"

					"strconv"

					"strings"

					"syscall"

					"time"

					"github.com/ergochat/readline"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"

					"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"

				)

				var (

					datasourceURL = flag.String("datasource.url", "http://localhost:9428/select/logsql/query", "URL for querying VictoriaLogs; "+

						"see https://docs.victoriametrics.com/victorialogs/querying/#querying-logs . See also -tail.url")

					tailURL = flag.String("tail.url", "", "URL for live tailing queries to VictoriaLogs; see https://docs.victoriametrics.com/victorialogs/querying/#live-tailing ."+

						"The url is automatically detected from -datasource.url by replacing /query with /tail at the end if -tail.url is empty")

					historyFile = flag.String("historyFile", "vlogscli-history", "Path to file with command history")

					header      = flagutil.NewArrayString("header", "Optional header to pass in request -datasource.url in the form 'HeaderName: value'")

					accountID   = flag.Int("accountID", 0, "Account ID to query; see https://docs.victoriametrics.com/victorialogs/#multitenancy")

					projectID   = flag.Int("projectID", 0, "Project ID to query; see https://docs.victoriametrics.com/victorialogs/#multitenancy")

				)

				const (

					firstLinePrompt = ";> "

					nextLinePrompt  = ""

				)

				func main() {

					// Write flags and help message to stdout, since it is easier to grep or pipe.

					flag.CommandLine.SetOutput(os.Stdout)

					flag.Usage = usage

					envflag.Parse()

					buildinfo.Init()

					logger.InitNoLogFlags()

					hes, err := parseHeaders(*header)

					if err != nil {

						fatalf("cannot parse -header command-line flag: %s", err)

					}

					headers = hes

					incompleteLine := ""

					cfg := &readline.Config{

						Prompt:                 firstLinePrompt,

						DisableAutoSaveHistory: true,

						Listener: func(line []rune, pos int, _ rune) ([]rune, int, bool) {

							incompleteLine = string(line)

							return line, pos, false

						},

					}

					rl, err := readline.NewFromConfig(cfg)

					if err != nil {

						fatalf("cannot initialize readline: %s", err)

					}

					fmt.Fprintf(rl, "sending queries to %s\n", *datasourceURL)

					runReadlineLoop(rl, &incompleteLine)

					if err := rl.Close(); err != nil {

						fatalf("cannot close readline: %s", err)

					}

				}

				func runReadlineLoop(rl *readline.Instance, incompleteLine *string) {

					historyLines, err := loadFromHistory(*historyFile)

					if err != nil {

						fatalf("cannot load query history: %s", err)

					}

					for _, line := range historyLines {

						if err := rl.SaveToHistory(line); err != nil {

							fatalf("cannot initialize query history: %s", err)

						}

					}

					outputMode := outputModeJSONMultiline

					s := ""

					for {

						line, err := rl.ReadLine()

						if err != nil {

							switch err {

							case io.EOF:

								if s != "" {

									// This is non-interactive query execution.

									executeQuery(context.Background(), rl, s, outputMode)

								}

								return

							case readline.ErrInterrupt:

								if s == "" && *incompleteLine == "" {

									fmt.Fprintf(rl, "interrupted\n")

									os.Exit(128 + int(syscall.SIGINT))

								}

								// Default value for Ctrl+C - clear the prompt and store the incompletely entered line into history

								s += *incompleteLine

								historyLines = pushToHistory(rl, historyLines, s)

								s = ""

								rl.SetPrompt(firstLinePrompt)

								continue

							default:

								fatalf("unexpected error in readline: %s", err)

							}

						}

						s += line

						if s == "" {

							// Skip empty lines

							continue

						}

						if isQuitCommand(s) {

							fmt.Fprintf(rl, "bye!\n")

							_ = pushToHistory(rl, historyLines, s)

							return

						}

						if isHelpCommand(s) {

							printCommandsHelp(rl)

							historyLines = pushToHistory(rl, historyLines, s)

							s = ""

							continue

						}

						if s == `\s` {

							fmt.Fprintf(rl, "singleline json output mode\n")

							outputMode = outputModeJSONSingleline

							historyLines = pushToHistory(rl, historyLines, s)

							s = ""

							continue

						}

						if s == `\m` {

							fmt.Fprintf(rl, "multiline json output mode\n")

							outputMode = outputModeJSONMultiline

							historyLines = pushToHistory(rl, historyLines, s)

							s = ""

							continue

						}

						if s == `\c` {

							fmt.Fprintf(rl, "compact output mode\n")

							outputMode = outputModeCompact

							historyLines = pushToHistory(rl, historyLines, s)

							s = ""

							continue

						}

						if s == `\logfmt` {

							fmt.Fprintf(rl, "logfmt output mode\n")

							outputMode = outputModeLogfmt

							historyLines = pushToHistory(rl, historyLines, s)

							s = ""

							continue

						}

						if line != "" && !strings.HasSuffix(line, ";") {

							// Assume the query is incomplete and allow the user finishing the query on the next line

							s += "\n"

							rl.SetPrompt(nextLinePrompt)

							continue

						}

						// Execute the query

						ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)

						executeQuery(ctx, rl, s, outputMode)

						cancel()

						historyLines = pushToHistory(rl, historyLines, s)

						s = ""

						rl.SetPrompt(firstLinePrompt)

					}

				}

				func pushToHistory(rl *readline.Instance, historyLines []string, s string) []string {

					s = strings.TrimSpace(s)

					if len(historyLines) == 0 || historyLines[len(historyLines)-1] != s {

						historyLines = append(historyLines, s)

						if len(historyLines) > 500 {

							historyLines = historyLines[len(historyLines)-500:]

						}

						if err := saveToHistory(*historyFile, historyLines); err != nil {

							fatalf("cannot save query history: %s", err)

						}

					}

					if err := rl.SaveToHistory(s); err != nil {

						fatalf("cannot update query history: %s", err)

					}

					return historyLines

				}

				func loadFromHistory(filePath string) ([]string, error) {

					data, err := os.ReadFile(filePath)

					if err != nil {

						if errors.Is(err, fs.ErrNotExist) {

							return nil, nil

						}

						return nil, err

					}

					linesQuoted := strings.Split(string(data), "\n")

					lines := make([]string, 0, len(linesQuoted))

					i := 0

					for _, lineQuoted := range linesQuoted {

						i++

						if lineQuoted == "" {

							continue

						}

						line, err := strconv.Unquote(lineQuoted)

						if err != nil {

							return nil, fmt.Errorf("cannot parse line #%d at %s: %w; line: [%s]", i, filePath, err, line)

						}

						lines = append(lines, line)

					}

					return lines, nil

				}

				func saveToHistory(filePath string, lines []string) error {

					linesQuoted := make([]string, len(lines))

					for i, line := range lines {

						lineQuoted := strconv.Quote(line)

						linesQuoted[i] = lineQuoted

					}

					data := strings.Join(linesQuoted, "\n")

					return os.WriteFile(filePath, []byte(data), 0600)

				}

				func isQuitCommand(s string) bool {

					switch s {

					case `\q`, "q", "quit", "exit":

						return true

					default:

						return false

					}

				}

				func isHelpCommand(s string) bool {

					switch s {

					case `\h`, "h", "help", "?":

						return true

					default:

						return false

					}

				}

				func printCommandsHelp(w io.Writer) {

					fmt.Fprintf(w, "%s", `List of available commands:

				\q - quit

				\h - show this help

				\s - singleline json output mode

				\m - multiline json output mode

				\c - compact output

				\logfmt - logfmt output mode

				\tail <query> - live tail <query> results

				`)

				}

				func executeQuery(ctx context.Context, output io.Writer, qStr string, outputMode outputMode) {

					if strings.HasPrefix(qStr, `\tail `) {

						tailQuery(ctx, output, qStr, outputMode)

						return

					}

					respBody := getQueryResponse(ctx, output, qStr, outputMode, *datasourceURL)

					if respBody == nil {

						return

					}

					defer func() {

						_ = respBody.Close()

					}()

					if err := readWithLess(respBody); err != nil {

						fmt.Fprintf(output, "error when reading query response: %s\n", err)

						return

					}

				}

				func tailQuery(ctx context.Context, output io.Writer, qStr string, outputMode outputMode) {

					qStr = strings.TrimPrefix(qStr, `\tail `)

					qURL, err := getTailURL()

					if err != nil {

						fmt.Fprintf(output, "%s\n", err)

						return

					}

					respBody := getQueryResponse(ctx, output, qStr, outputMode, qURL)

					if respBody == nil {

						return

					}

					defer func() {

						_ = respBody.Close()

					}()

					if _, err := io.Copy(output, respBody); err != nil {

						if !errors.Is(err, context.Canceled) && !isErrPipe(err) {

							fmt.Fprintf(output, "error when live tailing query response: %s\n", err)

						}

						fmt.Fprintf(output, "\n")

						return

					}

				}

				func getTailURL() (string, error) {

					if *tailURL != "" {

						return *tailURL, nil

					}

					u, err := url.Parse(*datasourceURL)

					if err != nil {

						return "", fmt.Errorf("cannot parse -datasource.url=%q: %w", *datasourceURL, err)

					}

					if !strings.HasSuffix(u.Path, "/query") {

						return "", fmt.Errorf("cannot find /query suffix in -datasource.url=%q", *datasourceURL)

					}

					u.Path = u.Path[:len(u.Path)-len("/query")] + "/tail"

					return u.String(), nil

				}

				func getQueryResponse(ctx context.Context, output io.Writer, qStr string, outputMode outputMode, qURL string) io.ReadCloser {

					// Parse the query and convert it to canonical view.

					qStr = strings.TrimSuffix(qStr, ";")

					q, err := logstorage.ParseQuery(qStr)

					if err != nil {

						fmt.Fprintf(output, "cannot parse query: %s\n", err)

						return nil

					}

					qStr = q.String()

					fmt.Fprintf(output, "executing [%s]...", qStr)

					// Prepare HTTP request for qURL

					args := make(url.Values)

					args.Set("query", qStr)

					data := strings.NewReader(args.Encode())

					req, err := http.NewRequestWithContext(ctx, "POST", qURL, data)

					if err != nil {

						panic(fmt.Errorf("BUG: cannot prepare request to server: %w", err))

					}

					req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

					for _, h := range headers {

						req.Header.Set(h.Name, h.Value)

					}

					req.Header.Set("AccountID", strconv.Itoa(*accountID))

					req.Header.Set("ProjectID", strconv.Itoa(*projectID))

					// Execute HTTP request at qURL

					startTime := time.Now()

					resp, err := httpClient.Do(req)

					queryDuration := time.Since(startTime)

					fmt.Fprintf(output, "; duration: %.3fs\n", queryDuration.Seconds())

					if err != nil {

						if errors.Is(err, context.Canceled) {

							fmt.Fprintf(output, "\n")

						} else {

							fmt.Fprintf(output, "cannot execute query: %s\n", err)

						}

						return nil

					}

					// Verify response code

					if resp.StatusCode != http.StatusOK {

						body, err := io.ReadAll(resp.Body)

						if err != nil {

							body = []byte(fmt.Sprintf("cannot read response body: %s", err))

						}

						fmt.Fprintf(output, "unexpected status code: %d; response body:\n%s\n", resp.StatusCode, body)

						return nil

					}

					// Prettify the response body

					jp := newJSONPrettifier(resp.Body, outputMode)

					return jp

				}

				var httpClient = &http.Client{}

				var headers []headerEntry

				type headerEntry struct {

					Name  string

					Value string

				}

				func parseHeaders(a []string) ([]headerEntry, error) {

					hes := make([]headerEntry, len(a))

					for i, s := range a {

						a := strings.SplitN(s, ":", 2)

						if len(a) != 2 {

							return nil, fmt.Errorf("cannot parse header=%q; it must contain at least one ':'; for example, 'Cookie: foo'", s)

						}

						hes[i] = headerEntry{

							Name:  strings.TrimSpace(a[0]),

							Value: strings.TrimSpace(a[1]),

						}

					}

					return hes, nil

				}

				func fatalf(format string, args ...any) {

					fmt.Fprintf(os.Stderr, format+"\n", args...)

					os.Exit(1)

				}

				func usage() {

					const s = `

				vlogscli is a command-line tool for querying VictoriaLogs.

				See the docs at https://docs.victoriametrics.com/victorialogs/querying/vlogscli/

				`

					flagutil.Usage(s)

				}

									
										299

app/vlselect/logsql/logsql.go
									
												View File
												
				@@ -5,7 +5,6 @@ import (

					"fmt"

					"math"

					"net/http"

					"slices"

					"sort"

					"strconv"

					"strings"

				@@ -45,7 +44,6 @@ func ProcessHitsRequest(ctx context.Context, w http.ResponseWriter, r *http.Requ

					}

					if step <= 0 {

						httpserver.Errorf(w, r, "'step' must be bigger than zero")

						return

					}

					// Obtain offset

				@@ -382,8 +380,6 @@ func ProcessStreamsRequest(ctx context.Context, w http.ResponseWriter, r *http.R

				}

				// ProcessLiveTailRequest processes live tailing request to /select/logsq/tail

				//

				// See https://docs.victoriametrics.com/victorialogs/querying/#live-tailing

				func ProcessLiveTailRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {

					liveTailRequests.Inc()

					defer liveTailRequests.Dec()

				@@ -394,9 +390,7 @@ func ProcessLiveTailRequest(ctx context.Context, w http.ResponseWriter, r *http.

						return

					}

					if !q.CanLiveTail() {

						httpserver.Errorf(w, r, "the query [%s] cannot be used in live tailing; "+

							"see https://docs.victoriametrics.com/victorialogs/querying/#live-tailing for details", q)

						return

						httpserver.Errorf(w, r, "the query [%s] cannot be used in live tailing; see https://docs.victoriametrics.com/victorialogs/querying/#live-tailing for details", q)

					}

					q.Optimize()

				@@ -419,17 +413,13 @@ func ProcessLiveTailRequest(ctx context.Context, w http.ResponseWriter, r *http.

					if !ok {

						logger.Panicf("BUG: it is expected that http.ResponseWriter (%T) supports http.Flusher interface", w)

					}

					qOrig := q

					for {

						start := end - tailOffsetNsecs

						end = time.Now().UnixNano()

						q = qOrig.Clone(end)

						q.AddTimeFilter(start, end)

						// q.Optimize() call is needed for converting '*' into filterNoop.

						// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785#issuecomment-2358547733

						q.Optimize()

						if err := vlstorage.RunQuery(ctxWithCancel, tenantIDs, q, tp.writeBlock); err != nil {

						qCopy := q.Clone()

						qCopy.AddTimeFilter(start, end)

						if err := vlstorage.RunQuery(ctxWithCancel, tenantIDs, qCopy, tp.writeBlock); err != nil {

							httpserver.Errorf(w, r, "cannot execute tail query [%s]: %s", q, err)

							return

						}

				@@ -570,212 +560,9 @@ func (tp *tailProcessor) getTailRows() ([][]logstorage.Field, error) {

					return tailRows, nil

				}

				// ProcessStatsQueryRangeRequest handles /select/logsql/stats_query_range request.

				//

				// See https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats

				func ProcessStatsQueryRangeRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {

					q, tenantIDs, err := parseCommonArgs(r)

					if err != nil {

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					// Obtain step

					stepStr := r.FormValue("step")

					if stepStr == "" {

						stepStr = "1d"

					}

					step, err := promutils.ParseDuration(stepStr)

					if err != nil {

						err = fmt.Errorf("cannot parse 'step' arg: %s", err)

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					if step <= 0 {

						err := fmt.Errorf("'step' must be bigger than zero")

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					// Obtain `by(...)` fields from the last `| stats` pipe in q.

					// Add `_time:step` to the `by(...)` list.

					byFields, err := q.GetStatsByFieldsAddGroupingByTime(int64(step))

					if err != nil {

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					q.Optimize()

					m := make(map[string]*statsSeries)

					var mLock sync.Mutex

					writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {

						clonedColumnNames := make([]string, len(columns))

						for i, c := range columns {

							clonedColumnNames[i] = strings.Clone(c.Name)

						}

						for i := range timestamps {

							timestamp := q.GetTimestamp()

							labels := make([]logstorage.Field, 0, len(byFields))

							for j, c := range columns {

								if c.Name == "_time" {

									nsec, ok := logstorage.TryParseTimestampRFC3339Nano(c.Values[i])

									if ok {

										timestamp = nsec

										continue

									}

								}

								if slices.Contains(byFields, c.Name) {

									labels = append(labels, logstorage.Field{

										Name:  clonedColumnNames[j],

										Value: strings.Clone(c.Values[i]),

									})

								}

							}

							var dst []byte

							for j, c := range columns {

								if !slices.Contains(byFields, c.Name) {

									name := clonedColumnNames[j]

									dst = dst[:0]

									dst = append(dst, name...)

									dst = logstorage.MarshalFieldsToJSON(dst, labels)

									key := string(dst)

									p := statsPoint{

										Timestamp: timestamp,

										Value:     strings.Clone(c.Values[i]),

									}

									mLock.Lock()

									ss := m[key]

									if ss == nil {

										ss = &statsSeries{

											key:    key,

											Name:   name,

											Labels: labels,

										}

										m[key] = ss

									}

									ss.Points = append(ss.Points, p)

									mLock.Unlock()

								}

							}

						}

					}

					if err := vlstorage.RunQuery(ctx, tenantIDs, q, writeBlock); err != nil {

						err = fmt.Errorf("cannot execute query [%s]: %s", q, err)

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					// Sort the collected stats by time

					rows := make([]*statsSeries, 0, len(m))

					for _, ss := range m {

						points := ss.Points

						sort.Slice(points, func(i, j int) bool {

							return points[i].Timestamp < points[j].Timestamp

						})

						rows = append(rows, ss)

					}

					sort.Slice(rows, func(i, j int) bool {

						return rows[i].key < rows[j].key

					})

					w.Header().Set("Content-Type", "application/json")

					WriteStatsQueryRangeResponse(w, rows)

				}

				type statsSeries struct {

					key string

					Name   string

					Labels []logstorage.Field

					Points []statsPoint

				}

				type statsPoint struct {

					Timestamp int64

					Value     string

				}

				// ProcessStatsQueryRequest handles /select/logsql/stats_query request.

				//

				// See https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats

				func ProcessStatsQueryRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {

					q, tenantIDs, err := parseCommonArgs(r)

					if err != nil {

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					// Obtain `by(...)` fields from the last `| stats` pipe in q.

					byFields, err := q.GetStatsByFields()

					if err != nil {

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					q.Optimize()

					var rows []statsRow

					var rowsLock sync.Mutex

					timestamp := q.GetTimestamp()

					writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {

						clonedColumnNames := make([]string, len(columns))

						for i, c := range columns {

							clonedColumnNames[i] = strings.Clone(c.Name)

						}

						for i := range timestamps {

							labels := make([]logstorage.Field, 0, len(byFields))

							for j, c := range columns {

								if slices.Contains(byFields, c.Name) {

									labels = append(labels, logstorage.Field{

										Name:  clonedColumnNames[j],

										Value: strings.Clone(c.Values[i]),

									})

								}

							}

							for j, c := range columns {

								if !slices.Contains(byFields, c.Name) {

									r := statsRow{

										Name:      clonedColumnNames[j],

										Labels:    labels,

										Timestamp: timestamp,

										Value:     strings.Clone(c.Values[i]),

									}

									rowsLock.Lock()

									rows = append(rows, r)

									rowsLock.Unlock()

								}

							}

						}

					}

					if err := vlstorage.RunQuery(ctx, tenantIDs, q, writeBlock); err != nil {

						err = fmt.Errorf("cannot execute query [%s]: %s", q, err)

						httpserver.SendPrometheusError(w, r, err)

						return

					}

					w.Header().Set("Content-Type", "application/json")

					WriteStatsQueryResponse(w, rows)

				}

				type statsRow struct {

					Name      string

					Labels    []logstorage.Field

					Timestamp int64

					Value     string

				}

				// ProcessQueryRequest handles /select/logsql/query request.

				//

				// See https://docs.victoriametrics.com/victorialogs/querying/#querying-logs

				// See https://docs.victoriametrics.com/victorialogs/querying/#http-api

				func ProcessQueryRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {

					q, tenantIDs, err := parseCommonArgs(r)

					if err != nil {

				@@ -850,7 +637,6 @@ func getLastNQueryResults(ctx context.Context, tenantIDs []logstorage.TenantID,

					limitUpper := 2 * limit

					q.AddPipeLimit(uint64(limitUpper))

					q.Optimize()

					rows, err := getQueryResultsWithLimit(ctx, tenantIDs, q, limitUpper)

					if err != nil {

						return nil, err

				@@ -861,62 +647,32 @@ func getLastNQueryResults(ctx context.Context, tenantIDs []logstorage.TenantID,

						return rows, nil

					}

					// Slow path - adjust time range for selecting up to limitUpper rows

					// Slow path - search for the time range containing up to limitUpper rows.

					start, end := q.GetFilterTimeRange()

					d := end/2 - start/2

					start += d

					qOrig := q

					for {

						timestamp := qOrig.GetTimestamp()

						q = qOrig.Clone(timestamp)

						q = qOrig.Clone()

						q.AddTimeFilter(start, end)

						// q.Optimize() call is needed for converting '*' into filterNoop.

						// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785#issuecomment-2358547733

						q.Optimize()

						rows, err := getQueryResultsWithLimit(ctx, tenantIDs, q, limitUpper)

						if err != nil {

							return nil, err

						}

						if d == 0 || start >= end {

							// The [start ... end] time range equals one nanosecond.

							// Just return up to limit rows.

							if len(rows) > limit {

								rows = rows[:limit]

							}

							return rows, nil

						}

						dLastBit := d & 1

						d /= 2

						if len(rows) >= limitUpper {

							// The number of found rows on the [start ... end] time range exceeds limitUpper,

							// so reduce the time range to [start+d ... end].

							start += d

							continue

						}

						if len(rows) >= limit {

							// The number of found rows is in the range [limit ... limitUpper).

							// This means that found rows contains the needed limit rows with the biggest timestamps.

						if len(rows) >= limit && len(rows) < limitUpper || d == 0 {

							rows = getLastNRows(rows, limit)

							return rows, nil

						}

						// The number of found rows on [start ... end] time range is below the limit.

						// This means the time range doesn't cover the needed logs, so it must be extended.

						if len(rows) == 0 {

							// The [start ... end] time range doesn't contain any rows, so change it to [start-d ... start).

							end = start - 1

							start -= d + dLastBit

							continue

						lastBit := d & 1

						d /= 2

						if len(rows) > limit {

							start += d

						} else {

							start -= d + lastBit

						}

						// The number of found rows on [start ... end] time range is bigger than 0 but smaller than limit.

						// Increase the time range to [start-d ... end].

						start -= d + dLastBit

					}

				}

				@@ -937,25 +693,20 @@ func getQueryResultsWithLimit(ctx context.Context, tenantIDs []logstorage.Tenant

					var rows []row

					var rowsLock sync.Mutex

					writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {

						clonedColumnNames := make([]string, len(columns))

						for i, c := range columns {

							clonedColumnNames[i] = strings.Clone(c.Name)

						}

						rowsLock.Lock()

						defer rowsLock.Unlock()

						for i, timestamp := range timestamps {

							fields := make([]logstorage.Field, len(columns))

							for j := range columns {

								f := &fields[j]

								f.Name = clonedColumnNames[j]

								f.Name = strings.Clone(columns[j].Name)

								f.Value = strings.Clone(columns[j].Values[i])

							}

							rowsLock.Lock()

							rows = append(rows, row{

								timestamp: timestamp,

								fields:    fields,

							})

							rowsLock.Unlock()

						}

						if len(rows) >= limit {

				@@ -977,23 +728,9 @@ func parseCommonArgs(r *http.Request) (*logstorage.Query, []logstorage.TenantID,

					}

					tenantIDs := []logstorage.TenantID{tenantID}

					// Parse optional time arg

					timestamp, okTime, err := getTimeNsec(r, "time")

					if err != nil {

						return nil, nil, err

					}

					if !okTime {

						// If time arg is missing, then evaluate query at the current timestamp

						timestamp = time.Now().UnixNano()

					}

					// decrease timestamp by one nanosecond in order to avoid capturing logs belonging

					// to the first nanosecond at the next period of time (month, week, day, hour, etc.)

					timestamp--

					// Parse query

					qStr := r.FormValue("query")

					q, err := logstorage.ParseQueryAtTimestamp(qStr, timestamp)

					q, err := logstorage.ParseQuery(qStr)

					if err != nil {

						return nil, nil, fmt.Errorf("cannot parse query [%s]: %s", qStr, err)

					}

35

app/vlselect/logsql/query_response.qtpl

View File

@@ -6,31 +6,15 @@
 // JSONRow creates JSON row from the given fields.
 {% func JSONRow(columns []logstorage.BlockColumn, rowIdx int) %}
 	{% code
 		i := 0
 		for i < len(columns) && columns[i].Values[rowIdx] == "" {
 			i++
 		}
 		columns = columns[i:]
 	%}
 	{% if len(columns) == 0 %}
 		{% return %}
 	{% endif %}
 	{
 	{% code	c := &columns[0] %}
 {
 	{% code c := &columns[0] %}
 	{%q= c.Name %}:{%q= c.Values[rowIdx] %}
 	{% code columns = columns[1:] %}
 	{% for colIdx := range columns %}
 		{% code
 			c := &columns[colIdx]
 			v := c.Values[rowIdx]
 		%}
 		{% if v == "" %}
 			{% continue %}
 		{% endif %}
 		{% code c := &columns[colIdx] %}
 		,{%q= c.Name %}:{%q= c.Values[rowIdx] %}
 	{% endfor %}
 	}{% newline %}
 }{% newline %}
 {% endfunc %}
 // JSONRows prints formatted rows
@@ -39,11 +23,7 @@
 		{% return %}
 	{% endif %}
 	{% for _, fields := range rows %}
 		{% code fields = logstorage.SkipLeadingFieldsWithoutValues(fields) %}
 		{% if len(fields) == 0 %}
 			{% continue %}
 		{% endif %}
 		{
 	{
 		{% if len(fields) > 0 %}
 			{% code
 				f := fields[0]
@@ -51,13 +31,10 @@
 			%}
 			{%q= f.Name %}:{%q= f.Value %}
 			{% for _, f := range fields %}
 				{% if f.Value == "" %}
 					{% continue %}
 				{% endif %}
 				,{%q= f.Name %}:{%q= f.Value %}
 			{% endfor %}
 		{% endif %}
 		}{% newline %}
 	}{% newline %}
 	{% endfor %}
 {% endfunc %}

									
										159

app/vlselect/logsql/query_response.qtpl.go
									
												View File
												
				@@ -26,176 +26,141 @@ var (

				//line app/vlselect/logsql/query_response.qtpl:8

				func StreamJSONRow(qw422016 *qt422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {

				//line app/vlselect/logsql/query_response.qtpl:10

					i := 0

					for i < len(columns) && columns[i].Values[rowIdx] == "" {

						i++

					}

					columns = columns[i:]

				//line app/vlselect/logsql/query_response.qtpl:16

					if len(columns) == 0 {

				//line app/vlselect/logsql/query_response.qtpl:17

						return

				//line app/vlselect/logsql/query_response.qtpl:18

					}

				//line app/vlselect/logsql/query_response.qtpl:18

				//line app/vlselect/logsql/query_response.qtpl:8

					qw422016.N().S(`{`)

				//line app/vlselect/logsql/query_response.qtpl:20

				//line app/vlselect/logsql/query_response.qtpl:10

					c := &columns[0]

				//line app/vlselect/logsql/query_response.qtpl:21

				//line app/vlselect/logsql/query_response.qtpl:11

					qw422016.N().Q(c.Name)

				//line app/vlselect/logsql/query_response.qtpl:21

				//line app/vlselect/logsql/query_response.qtpl:11

					qw422016.N().S(`:`)

				//line app/vlselect/logsql/query_response.qtpl:21

				//line app/vlselect/logsql/query_response.qtpl:11

					qw422016.N().Q(c.Values[rowIdx])

				//line app/vlselect/logsql/query_response.qtpl:22

				//line app/vlselect/logsql/query_response.qtpl:12

					columns = columns[1:]

				//line app/vlselect/logsql/query_response.qtpl:23

				//line app/vlselect/logsql/query_response.qtpl:13

					for colIdx := range columns {

				//line app/vlselect/logsql/query_response.qtpl:25

				//line app/vlselect/logsql/query_response.qtpl:14

						c := &columns[colIdx]

						v := c.Values[rowIdx]

				//line app/vlselect/logsql/query_response.qtpl:28

						if v == "" {

				//line app/vlselect/logsql/query_response.qtpl:29

							continue

				//line app/vlselect/logsql/query_response.qtpl:30

						}

				//line app/vlselect/logsql/query_response.qtpl:30

				//line app/vlselect/logsql/query_response.qtpl:14

						qw422016.N().S(`,`)

				//line app/vlselect/logsql/query_response.qtpl:31

				//line app/vlselect/logsql/query_response.qtpl:15

						qw422016.N().Q(c.Name)

				//line app/vlselect/logsql/query_response.qtpl:31

				//line app/vlselect/logsql/query_response.qtpl:15

						qw422016.N().S(`:`)

				//line app/vlselect/logsql/query_response.qtpl:31

				//line app/vlselect/logsql/query_response.qtpl:15

						qw422016.N().Q(c.Values[rowIdx])

				//line app/vlselect/logsql/query_response.qtpl:32

				//line app/vlselect/logsql/query_response.qtpl:16

					}

				//line app/vlselect/logsql/query_response.qtpl:32

				//line app/vlselect/logsql/query_response.qtpl:16

					qw422016.N().S(`}`)

				//line app/vlselect/logsql/query_response.qtpl:33

				//line app/vlselect/logsql/query_response.qtpl:17

					qw422016.N().S(`

				`)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

				}

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

				func WriteJSONRow(qq422016 qtio422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					StreamJSONRow(qw422016, columns, rowIdx)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

				}

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

				func JSONRow(columns []logstorage.BlockColumn, rowIdx int) string {

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					WriteJSONRow(qb422016, columns, rowIdx)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

					return qs422016

				//line app/vlselect/logsql/query_response.qtpl:34

				//line app/vlselect/logsql/query_response.qtpl:18

				}

				// JSONRows prints formatted rows

				//line app/vlselect/logsql/query_response.qtpl:37

				//line app/vlselect/logsql/query_response.qtpl:21

				func StreamJSONRows(qw422016 *qt422016.Writer, rows [][]logstorage.Field) {

				//line app/vlselect/logsql/query_response.qtpl:38

				//line app/vlselect/logsql/query_response.qtpl:22

					if len(rows) == 0 {

				//line app/vlselect/logsql/query_response.qtpl:39

				//line app/vlselect/logsql/query_response.qtpl:23

						return

				//line app/vlselect/logsql/query_response.qtpl:40

				//line app/vlselect/logsql/query_response.qtpl:24

					}

				//line app/vlselect/logsql/query_response.qtpl:41

				//line app/vlselect/logsql/query_response.qtpl:25

					for _, fields := range rows {

				//line app/vlselect/logsql/query_response.qtpl:42

						fields = logstorage.SkipLeadingFieldsWithoutValues(fields)

				//line app/vlselect/logsql/query_response.qtpl:43

						if len(fields) == 0 {

				//line app/vlselect/logsql/query_response.qtpl:44

							continue

				//line app/vlselect/logsql/query_response.qtpl:45

						}

				//line app/vlselect/logsql/query_response.qtpl:45

				//line app/vlselect/logsql/query_response.qtpl:25

						qw422016.N().S(`{`)

				//line app/vlselect/logsql/query_response.qtpl:47

				//line app/vlselect/logsql/query_response.qtpl:27

						if len(fields) > 0 {

				//line app/vlselect/logsql/query_response.qtpl:49

				//line app/vlselect/logsql/query_response.qtpl:29

							f := fields[0]

							fields = fields[1:]

				//line app/vlselect/logsql/query_response.qtpl:52

				//line app/vlselect/logsql/query_response.qtpl:32

							qw422016.N().Q(f.Name)

				//line app/vlselect/logsql/query_response.qtpl:52

				//line app/vlselect/logsql/query_response.qtpl:32

							qw422016.N().S(`:`)

				//line app/vlselect/logsql/query_response.qtpl:52

				//line app/vlselect/logsql/query_response.qtpl:32

							qw422016.N().Q(f.Value)

				//line app/vlselect/logsql/query_response.qtpl:53

				//line app/vlselect/logsql/query_response.qtpl:33

							for _, f := range fields {

				//line app/vlselect/logsql/query_response.qtpl:54

								if f.Value == "" {

				//line app/vlselect/logsql/query_response.qtpl:55

									continue

				//line app/vlselect/logsql/query_response.qtpl:56

								}

				//line app/vlselect/logsql/query_response.qtpl:56

				//line app/vlselect/logsql/query_response.qtpl:33

								qw422016.N().S(`,`)

				//line app/vlselect/logsql/query_response.qtpl:57

				//line app/vlselect/logsql/query_response.qtpl:34

								qw422016.N().Q(f.Name)

				//line app/vlselect/logsql/query_response.qtpl:57

				//line app/vlselect/logsql/query_response.qtpl:34

								qw422016.N().S(`:`)

				//line app/vlselect/logsql/query_response.qtpl:57

				//line app/vlselect/logsql/query_response.qtpl:34

								qw422016.N().Q(f.Value)

				//line app/vlselect/logsql/query_response.qtpl:58

				//line app/vlselect/logsql/query_response.qtpl:35

							}

				//line app/vlselect/logsql/query_response.qtpl:59

				//line app/vlselect/logsql/query_response.qtpl:36

						}

				//line app/vlselect/logsql/query_response.qtpl:59

				//line app/vlselect/logsql/query_response.qtpl:36

						qw422016.N().S(`}`)

				//line app/vlselect/logsql/query_response.qtpl:60

				//line app/vlselect/logsql/query_response.qtpl:37

						qw422016.N().S(`

				`)

				//line app/vlselect/logsql/query_response.qtpl:61

				//line app/vlselect/logsql/query_response.qtpl:38

					}

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

				}

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

				func WriteJSONRows(qq422016 qtio422016.Writer, rows [][]logstorage.Field) {

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					StreamJSONRows(qw422016, rows)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

				}

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

				func JSONRows(rows [][]logstorage.Field) string {

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					WriteJSONRows(qb422016, rows)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

					return qs422016

				//line app/vlselect/logsql/query_response.qtpl:62

				//line app/vlselect/logsql/query_response.qtpl:39

				}

52

app/vlselect/logsql/stats_query_range_response.qtpl

View File

@@ -1,52 +0,0 @@
 {% stripspace %}
 // StatsQueryRangeResponse generates response for /select/logsql/stats_query_range
 {% func StatsQueryRangeResponse(rows []*statsSeries) %}
 {
 	"status":"success",
 	"data":{
 		"resultType":"matrix",
 		"result":[
 			{% if len(rows) > 0 %}
 				{%= formatStatsSeries(rows[0]) %}
 				{% code rows = rows[1:] %}
 				{% for i := range rows %}
 					,{%= formatStatsSeries(rows[i]) %}
 				{% endfor %}
 			{% endif %}
 		]
 	}
 }
 {% endfunc %}
 {% func formatStatsSeries(ss *statsSeries) %}
 {
 	"metric":{
 		"__name__":{%q= ss.Name %}
 		{% if len(ss.Labels) > 0 %}
 			{% for _, label := range ss.Labels %}
 				,{%q= label.Name %}:{%q= label.Value %}
 			{% endfor %}
 		{% endif %}
 	},
 	"values":[
 		{% code points := ss.Points %}
 		{% if len(points) > 0 %}
 			{%= formatStatsPoint(&points[0]) %}
 			{% code points = points[1:] %}
 			{% for i := range points %}
 				,{%= formatStatsPoint(&points[i]) %}
 			{% endfor %}
 		{% endif %}
 	]
 }
 {% endfunc %}
 {% func formatStatsPoint(p *statsPoint) %}
 [
 	{%f= float64(p.Timestamp)/1e9 %},
 	{%q= p.Value %}
 ]
 {% endfunc %}
 {% endstripspace %}

									
										188

app/vlselect/logsql/stats_query_range_response.qtpl.go
									
												View File
											
				@@ -1,188 +0,0 @@

				// Code generated by qtc from "stats_query_range_response.qtpl". DO NOT EDIT.

				// See https://github.com/valyala/quicktemplate for details.

				// StatsQueryRangeResponse generates response for /select/logsql/stats_query_range

				//line app/vlselect/logsql/stats_query_range_response.qtpl:4

				package logsql

				//line app/vlselect/logsql/stats_query_range_response.qtpl:4

				import (

					qtio422016 "io"

					qt422016 "github.com/valyala/quicktemplate"

				)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:4

				var (

					_ = qtio422016.Copy

					_ = qt422016.AcquireByteBuffer

				)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:4

				func StreamStatsQueryRangeResponse(qw422016 *qt422016.Writer, rows []*statsSeries) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:4

					qw422016.N().S(`{"status":"success","data":{"resultType":"matrix","result":[`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:10

					if len(rows) > 0 {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:11

						streamformatStatsSeries(qw422016, rows[0])

				//line app/vlselect/logsql/stats_query_range_response.qtpl:12

						rows = rows[1:]

				//line app/vlselect/logsql/stats_query_range_response.qtpl:13

						for i := range rows {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:13

							qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:14

							streamformatStatsSeries(qw422016, rows[i])

				//line app/vlselect/logsql/stats_query_range_response.qtpl:15

						}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:16

					}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:16

					qw422016.N().S(`]}}`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

				func WriteStatsQueryRangeResponse(qq422016 qtio422016.Writer, rows []*statsSeries) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					StreamStatsQueryRangeResponse(qw422016, rows)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

				func StatsQueryRangeResponse(rows []*statsSeries) string {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					WriteStatsQueryRangeResponse(qb422016, rows)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

					return qs422016

				//line app/vlselect/logsql/stats_query_range_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:22

				func streamformatStatsSeries(qw422016 *qt422016.Writer, ss *statsSeries) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:22

					qw422016.N().S(`{"metric":{"__name__":`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:25

					qw422016.N().Q(ss.Name)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:26

					if len(ss.Labels) > 0 {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:27

						for _, label := range ss.Labels {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:27

							qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:28

							qw422016.N().Q(label.Name)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:28

							qw422016.N().S(`:`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:28

							qw422016.N().Q(label.Value)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:29

						}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:30

					}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:30

					qw422016.N().S(`},"values":[`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:33

					points := ss.Points

				//line app/vlselect/logsql/stats_query_range_response.qtpl:34

					if len(points) > 0 {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:35

						streamformatStatsPoint(qw422016, &points[0])

				//line app/vlselect/logsql/stats_query_range_response.qtpl:36

						points = points[1:]

				//line app/vlselect/logsql/stats_query_range_response.qtpl:37

						for i := range points {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:37

							qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:38

							streamformatStatsPoint(qw422016, &points[i])

				//line app/vlselect/logsql/stats_query_range_response.qtpl:39

						}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:40

					}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:40

					qw422016.N().S(`]}`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

				func writeformatStatsSeries(qq422016 qtio422016.Writer, ss *statsSeries) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					streamformatStatsSeries(qw422016, ss)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

				func formatStatsSeries(ss *statsSeries) string {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					writeformatStatsSeries(qb422016, ss)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

					return qs422016

				//line app/vlselect/logsql/stats_query_range_response.qtpl:43

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:45

				func streamformatStatsPoint(qw422016 *qt422016.Writer, p *statsPoint) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:45

					qw422016.N().S(`[`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:47

					qw422016.N().F(float64(p.Timestamp) / 1e9)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:47

					qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:48

					qw422016.N().Q(p.Value)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:48

					qw422016.N().S(`]`)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

				func writeformatStatsPoint(qq422016 qtio422016.Writer, p *statsPoint) {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					streamformatStatsPoint(qw422016, p)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

				}

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

				func formatStatsPoint(p *statsPoint) string {

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					writeformatStatsPoint(qb422016, p)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

					return qs422016

				//line app/vlselect/logsql/stats_query_range_response.qtpl:50

				}

36

app/vlselect/logsql/stats_query_response.qtpl

View File

@@ -1,36 +0,0 @@
 {% stripspace %}
 // StatsQueryResponse generates response for /select/logsql/stats_query
 {% func StatsQueryResponse(rows []statsRow) %}
 {
 	"status":"success",
 	"data":{
 		"resultType":"vector",
 		"result":[
 			{% if len(rows) > 0 %}
 				{%= formatStatsRow(&rows[0]) %}
 				{% code rows = rows[1:] %}
 				{% for i := range rows %}
 					,{%= formatStatsRow(&rows[i]) %}
 				{% endfor %}
 			{% endif %}
 		]
 	}
 }
 {% endfunc %}
 {% func formatStatsRow(r *statsRow) %}
 {
 	"metric":{
 		"__name__":{%q= r.Name %}
 		{% if len(r.Labels) > 0 %}
 			{% for _, label := range r.Labels %}
 				,{%q= label.Name %}:{%q= label.Value %}
 			{% endfor %}
 		{% endif %}
 	},
 	"value":[{%f= float64(r.Timestamp)/1e9 %},{%q= r.Value %}]
 }
 {% endfunc %}
 {% endstripspace %}

									
										133

app/vlselect/logsql/stats_query_response.qtpl.go
									
												View File
											
				@@ -1,133 +0,0 @@

				// Code generated by qtc from "stats_query_response.qtpl". DO NOT EDIT.

				// See https://github.com/valyala/quicktemplate for details.

				// StatsQueryResponse generates response for /select/logsql/stats_query

				//line app/vlselect/logsql/stats_query_response.qtpl:4

				package logsql

				//line app/vlselect/logsql/stats_query_response.qtpl:4

				import (

					qtio422016 "io"

					qt422016 "github.com/valyala/quicktemplate"

				)

				//line app/vlselect/logsql/stats_query_response.qtpl:4

				var (

					_ = qtio422016.Copy

					_ = qt422016.AcquireByteBuffer

				)

				//line app/vlselect/logsql/stats_query_response.qtpl:4

				func StreamStatsQueryResponse(qw422016 *qt422016.Writer, rows []statsRow) {

				//line app/vlselect/logsql/stats_query_response.qtpl:4

					qw422016.N().S(`{"status":"success","data":{"resultType":"vector","result":[`)

				//line app/vlselect/logsql/stats_query_response.qtpl:10

					if len(rows) > 0 {

				//line app/vlselect/logsql/stats_query_response.qtpl:11

						streamformatStatsRow(qw422016, &rows[0])

				//line app/vlselect/logsql/stats_query_response.qtpl:12

						rows = rows[1:]

				//line app/vlselect/logsql/stats_query_response.qtpl:13

						for i := range rows {

				//line app/vlselect/logsql/stats_query_response.qtpl:13

							qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_response.qtpl:14

							streamformatStatsRow(qw422016, &rows[i])

				//line app/vlselect/logsql/stats_query_response.qtpl:15

						}

				//line app/vlselect/logsql/stats_query_response.qtpl:16

					}

				//line app/vlselect/logsql/stats_query_response.qtpl:16

					qw422016.N().S(`]}}`)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_response.qtpl:20

				func WriteStatsQueryResponse(qq422016 qtio422016.Writer, rows []statsRow) {

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					StreamStatsQueryResponse(qw422016, rows)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_response.qtpl:20

				func StatsQueryResponse(rows []statsRow) string {

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					WriteStatsQueryResponse(qb422016, rows)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:20

					return qs422016

				//line app/vlselect/logsql/stats_query_response.qtpl:20

				}

				//line app/vlselect/logsql/stats_query_response.qtpl:22

				func streamformatStatsRow(qw422016 *qt422016.Writer, r *statsRow) {

				//line app/vlselect/logsql/stats_query_response.qtpl:22

					qw422016.N().S(`{"metric":{"__name__":`)

				//line app/vlselect/logsql/stats_query_response.qtpl:25

					qw422016.N().Q(r.Name)

				//line app/vlselect/logsql/stats_query_response.qtpl:26

					if len(r.Labels) > 0 {

				//line app/vlselect/logsql/stats_query_response.qtpl:27

						for _, label := range r.Labels {

				//line app/vlselect/logsql/stats_query_response.qtpl:27

							qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_response.qtpl:28

							qw422016.N().Q(label.Name)

				//line app/vlselect/logsql/stats_query_response.qtpl:28

							qw422016.N().S(`:`)

				//line app/vlselect/logsql/stats_query_response.qtpl:28

							qw422016.N().Q(label.Value)

				//line app/vlselect/logsql/stats_query_response.qtpl:29

						}

				//line app/vlselect/logsql/stats_query_response.qtpl:30

					}

				//line app/vlselect/logsql/stats_query_response.qtpl:30

					qw422016.N().S(`},"value":[`)

				//line app/vlselect/logsql/stats_query_response.qtpl:32

					qw422016.N().F(float64(r.Timestamp) / 1e9)

				//line app/vlselect/logsql/stats_query_response.qtpl:32

					qw422016.N().S(`,`)

				//line app/vlselect/logsql/stats_query_response.qtpl:32

					qw422016.N().Q(r.Value)

				//line app/vlselect/logsql/stats_query_response.qtpl:32

					qw422016.N().S(`]}`)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

				}

				//line app/vlselect/logsql/stats_query_response.qtpl:34

				func writeformatStatsRow(qq422016 qtio422016.Writer, r *statsRow) {

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					qw422016 := qt422016.AcquireWriter(qq422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					streamformatStatsRow(qw422016, r)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					qt422016.ReleaseWriter(qw422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

				}

				//line app/vlselect/logsql/stats_query_response.qtpl:34

				func formatStatsRow(r *statsRow) string {

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					qb422016 := qt422016.AcquireByteBuffer()

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					writeformatStatsRow(qb422016, r)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					qs422016 := string(qb422016.B)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					qt422016.ReleaseByteBuffer(qb422016)

				//line app/vlselect/logsql/stats_query_response.qtpl:34

					return qs422016

				//line app/vlselect/logsql/stats_query_response.qtpl:34

				}

									
										10

app/vlselect/main.go
									
												View File
												
				@@ -193,14 +193,6 @@ func processSelectRequest(ctx context.Context, w http.ResponseWriter, r *http.Re

						logsqlQueryRequests.Inc()

						logsql.ProcessQueryRequest(ctx, w, r)

						return true

					case "/select/logsql/stats_query":

						logsqlStatsQueryRequests.Inc()

						logsql.ProcessStatsQueryRequest(ctx, w, r)

						return true

					case "/select/logsql/stats_query_range":

						logsqlStatsQueryRangeRequests.Inc()

						logsql.ProcessStatsQueryRangeRequest(ctx, w, r)

						return true

					case "/select/logsql/stream_field_names":

						logsqlStreamFieldNamesRequests.Inc()

						logsql.ProcessStreamFieldNamesRequest(ctx, w, r)

				@@ -240,8 +232,6 @@ var (

					logsqlFieldValuesRequests       = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/field_values"}`)

					logsqlHitsRequests              = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/hits"}`)

					logsqlQueryRequests             = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/query"}`)

					logsqlStatsQueryRequests        = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stats_query"}`)

					logsqlStatsQueryRangeRequests   = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stats_query_range"}`)

					logsqlStreamFieldNamesRequests  = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_field_names"}`)

					logsqlStreamFieldValuesRequests = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_field_values"}`)

					logsqlStreamIDsRequests         = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_ids"}`)

BIN
app/vlselect/vmui/apple-touch-icon.png Normal file

View File

Binary file not shown.

After

Width: | Height: | Size: 14 KiB

									
										12

app/vlselect/vmui/asset-manifest.json
									
												View File
												
				@@ -1,13 +1,13 @@

				{

				  "files": {

				    "main.css": "./static/css/main.faf86aa5.css",

				    "main.js": "./static/js/main.2810cc52.js",

				    "static/js/685.f772060c.chunk.js": "./static/js/685.f772060c.chunk.js",

				    "static/media/MetricsQL.md": "./static/media/MetricsQL.a00044c91d9781cf8557.md",

				    "main.css": "./static/css/main.1041c3d4.css",

				    "main.js": "./static/js/main.8451d9b9.js",

				    "static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",

				    "static/media/MetricsQL.md": "./static/media/MetricsQL.d46c42c8e891f06298c4.md",

				    "index.html": "./index.html"

				  },

				  "entrypoints": [

				    "static/css/main.faf86aa5.css",

				    "static/js/main.2810cc52.js"

				    "static/css/main.1041c3d4.css",

				    "static/js/main.8451d9b9.js"

				  ]

				}

									
										5

app/vlselect/vmui/config.json
									
												View File
											
				@@ -1,5 +0,0 @@

				{

				  "license": {

				    "type": "opensource"

				  }

				}

BIN
app/vlselect/vmui/favicon-32x32.png Normal file

View File

Binary file not shown.

After

Width: | Height: | Size: 1.6 KiB

1

app/vlselect/vmui/favicon.svg

View File

				`@@ -1 +0,0 @@`
				<svg width="48" height="48" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M24.5475 0C10.3246.0265251 1.11379 3.06365 4.40623 6.10077c0 0 12.32997 11.23333 16.58217 14.84083.8131.6896 2.1728 1.1936 3.5191 1.2201h.1199c1.3463-.0265 2.706-.5305 3.5191-1.2201 4.2522-3.5942 16.5422-14.84083 16.5422-14.84083C48.0478 3.06365 38.8636.0265251 24.6674 0" fill="#020202"/><path d="M28.1579 27.0159c-.8131.6896-2.1728 1.1936-3.5191 1.2201h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2201-2.9725-2.5067-13.35639-11.87-17.26201-15.3979v5.4112c0 .5968.22661 1.3793.6265 1.7506C7.00358 21.1936 17.2675 30.5437 20.9731 33.6737c.8132.6896 2.1728 1.1936 3.5191 1.2201h.12c1.3463-.0265 2.7059-.5305 3.519-1.2201 3.679-3.13 13.9429-12.4536 16.6089-14.8939.4132-.3713.6265-1.1538.6265-1.7506V11.618c-3.9323 3.5411-14.3162 12.931-17.2354 15.3979h.0267Z" fill="#020202"/><path d="M28.1579 39.748c-.8131.6897-2.1728 1.1937-3.5191 1.2202h-.12c-1.3463-.0265-2.7059-.5305-3.519-1.2202-2.9725-2.4933-13.35639-11.8567-17.26201-15.3978v5.4111c0 .5969.22661 1.3793.6265 1.7507C7.00358 33.9258 17.2675 43.2759 20.9731 46.4058c.8132.6897 2.1728 1.1937 3.5191 1.2202h.12c1.3463-.0265 2.7059-.5305 3.519-1.2202 3.679-3.1299 13.9429-12.4535 16.6089-14.8938.4132-.3714.6265-1.1538.6265-1.7507v-5.4111c-3.9323 3.5411-14.3162 12.931-17.2354 15.3978h.0267Z" fill="#020202"/></svg>

Before

Width: | Height: | Size: 1.3 KiB

									
										2

app/vlselect/vmui/index.html
									
												View File
												
				@@ -1 +1 @@

				<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.svg"/><link rel="apple-touch-icon" href="./favicon.svg"/><link rel="mask-icon" href="./favicon.svg" color="#000000"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="Explore your log data with VictoriaLogs UI"/><link rel="manifest" href="./manifest.json"/><title>UI for VictoriaLogs</title><meta name="twitter:card" content="summary"><meta name="twitter:title" content="UI for VictoriaLogs"><meta name="twitter:site" content="@https://victoriametrics.com/products/victorialogs/"><meta name="twitter:description" content="Explore your log data with VictoriaLogs UI"><meta name="twitter:image" content="./preview.jpg"><meta property="og:type" content="website"><meta property="og:title" content="UI for VictoriaLogs"><meta property="og:url" content="https://victoriametrics.com/products/victorialogs/"><meta property="og:description" content="Explore your log data with VictoriaLogs UI"><script defer="defer" src="./static/js/main.2810cc52.js"></script><link href="./static/css/main.faf86aa5.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

				<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><script src="./dashboards/index.js" type="module"></script><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.8451d9b9.js"></script><link href="./static/css/main.1041c3d4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

									
										15

app/vlselect/vmui/manifest.json
									
												View File
												
				@@ -1,11 +1,16 @@

				{

				  "short_name": "vmui",

				  "name": "vmui",

				  "short_name": "Victoria Metrics UI",

				  "name": "Victoria Metrics UI is a metric explorer for Victoria Metrics",

				  "icons": [

				    {

				      "src": "favicon.svg",

				      "sizes": "any",

				      "type": "image/svg+xml"

				      "src": "favicon-32x32.png",

				      "sizes": "32x32",

				      "type": "image/png"

				    },

				    {

				      "src": "apple-touch-icon.png",

				      "type": "image/png",

				      "sizes": "192x192"

				    }

				  ],

				  "start_url": ".",

BIN
app/vlselect/vmui/preview.jpg

View File

Binary file not shown.

Before

Width: | Height: | Size: 6.5 KiB

After

Width: | Height: | Size: 67 KiB