lib/workingsetcache: properly count cache metrics during rotation

Previously expirationWatcher and cacheSizeWatcher may update prev and
curr cache during metrics collection. It incorrectly reports metrics and
may confuse users.

 As an example, expirationWatcher sets currCache to the curr and prev
values during cache rotation. It messes metrics and reports high cache
hit misses. Which in reality doesn't exist. Cache requests metric is
doubled.

 This commit serializes metrics update with mutex lock. Since metrics
collection is fast operation and it doesn't lock on Get and Set
operations, it should bring any delays into storage
data processing.

docs/victoriametrics/changelog/CHANGELOG.md

@@ -28,6 +28,8 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
 
 * FEATURE: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): stream responses from backends to clients without delays. Previously the backend data could be buffered at `vmauth` side for indefinite amounts of time. This was preventing from using `vmauth` for streaming the data from backends in [live tailing mode](https://docs.victoriametrics.com/victorialogs/querying/#live-tailing). See [VictoriaLogs#667](https://github.com/VictoriaMetrics/VictoriaLogs/issues/667).
 
+* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly report `workingsetcache` metrics during cache rotation. See this PR []() for details.
+
 
 ## [v1.126.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.126.0)
 

lib/workingsetcache/cache.go

@@ -62,7 +62,7 @@ type Cache struct {
 	maxBytes int
 
 	// mu serializes access to curr, prev and mode
-	// in expirationWatcher, prevCacheWatcher and cacheSizeWatcher.
+	// in expirationWatcher, prevCacheWatcher,cacheSizeWatcher and UpdateStats.
 	mu sync.Mutex
 
 	wg     sync.WaitGroup
@@ -362,6 +362,9 @@ func (c *Cache) Reset() {
 
 // UpdateStats updates fcs with cache stats.
 func (c *Cache) UpdateStats(fcs *fastcache.Stats) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
 	updateCacheStatsHistory(fcs, &c.csHistory)
 
 	var cs fastcache.Stats