[BUG] ft.aggregate slowdown with high frequency updates about redisearch

[tommey]

We have also encountered a memory issue, causing replicas to slowly use up all configured memory amount that wasn't the case before with the regular redis server, without modules and no RediSearch indexing, so now we opted for a custom LUA Redis Function to do the aggregation, which calls KEYS prefix*, then loads the HashMaps with HGETALL, does the aggregation and returns an Array reply with Array items consisting of the tag and 4 counters.

The service performs at a stable 100ms response time, which is 2-3 times worse than it would be with FT.AGGREGATE, if it wouldn't slow down as max_doc_id increases.

I have looked at the BITFIELD type, which would be another way to implement this, but that would raise additional problems to solve around consistency, reading back data in pipelined workflows, in a use-case where everything changes every second.

[liwensen]

I had the same problem with the latest version(redis/redis-stack:7.2.0-v9):
Some simple JSON data (1K) was created with only 1 record.
Create an index (idx:Test) and then query the data through the index.
FT.SEARCH idx:Test *

(1) Queries are very fast at first.
(2) After trying to update about 1 million times, I started to stop updating the data and then query the data, which takes 200-400ms.
(3) I tried to delete the index and then re-create the index, which was to query the data again very quickly.
FT.INFO idx:Test
FT.DROPINDEX idx:Test