Comments (5)
Repost observation and experiments done by @MrCroxx for better visibility:
There are three observations:
- uneven data among different epochs when that epoch gets evicted.
-
a single eviction of multiple epochs and the size of data evicted is huge
-
a single eviction of a single epoch and the size of data evicted is huge
For phenomena 1 and 2, there was a speculation that each time an entry is accessed, the entry's epoch is increased, causing older epoch data to gradually decrease, leading to evicting data with fewer older epochs first. After evicting several epochs and finding that memory did not decrease, a more aggressive exponential evicting of epochs began, ultimately resulting in the eviction of data from epochs with a larger amount of newer data.
This speculation may explain the issue when the barrier frequency is high. However, for phenomenon 3 observed in the 10-second barrier experiment mentioned above, it seems more like there are too few epochs to adequately partition the eviction scope.
In the future, I will try other strategies such as incorporating epochs and quotas, using quotas only without epochs, or exploring alternative strategies.
from risingwave.
The evicted bytes by epoch (summed by all LRUs, 1s barrier):
https://1drv.ms/x/s!AiJJmrmsw_N2mxGoFZbGAYSRGstJ?e=B76XvA
from risingwave.
The metrics mentioned in #14797 (comment) is somehow expected under our current implementation of LRU memory manager.
The root cause is because the feedback loop between LRU watermark and actual usage is much slower than 1 second (the default running interval of the memory policy). The memory manager assumes its feedback at previous run has been reflected in the current memory usage, but it's not, actually.
To get rid of this assumption, we have discussed on several ideas. For example, we may let each streaming executor report its LRU cache memory usage by epoch, so that the memory manager can find out the best LRU watermark accordingly.
from risingwave.
TPC-H q4: #14811 (comment)
has a very similar observation
Edit:
q4 is not affected by eviction though as it does not need a very big cache and its cache miss ops/rate is very low
But q20 does get affected by this as shown at #14797 (comment)
from risingwave.
TPC-H q17: #14799 (comment) also has a very similar observation
from risingwave.
Related Issues (20)
- bug: main-cron e2e test failed caused by over window & reversed iteration
- sink to BigQuery error: `Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.` HOT 2
- Discussion: Support `explain(terse)` for queries
- Support decouple_commit for all olap sinks ? HOT 1
- set_config() doesn't accept parameters in the extended mode
- `BatchSort` gets dropped from an inner batch query HOT 1
- bump the mysql java library to the latest version
- feat(diagnosis): Print stacktrace of JVM along with await tree dump
- bug: wrong append_only property derivation on temporal join
- Bug: risedev program exited with 134 HOT 1
- cherrypick fix(stream): fix temporal join append only property (#17239) to branch release-1.9
- support dummy cast to REGOPER
- support pg_index_column_has_property
- check cargo workspace package version during CI to avoid version regression
- pg_index does not include primary keys HOT 3
- support passing `null` as a column constraint
- feat(source): check schema registry subject name before deserialization
- udf: `drop function` should perform dependency check HOT 5
- feat(source): protobuf field presence HOT 1
- arrangement backfill can be slow when there are consecutive tombstones in upstream table
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from risingwave.