Comments (2)
Compressing on a block-by-block basis seems like a natural approach, and is an approach utilized by other storage libraries like {go}leveldb, rocksdb, couchstore, etc.
One issue is the current segment implementation of mossStore doesn't even have a block abstraction. :-/ So a new kind of segment implementation is probably needed. With recent discussions with @mschoch on supporting additional kinds of segment implementations, this might not be too far out of bounds.
One thought would be that a new "compressing" segment kind might be engaged during compaction. The compactWriter{} could buffer up a bunch of mutations and then write out snappy compressed buf blocks, probably near or related to codepaths here...
https://github.com/couchbase/moss/blob/master/store_compact.go#L332
But, the kvs data might stay uncompressed.
When there's a binary search through the kvs array, the new segment kind would have to round down to the nearest compressed buf block, uncompress the buf block (where the uncompressed bytes can perhaps be cached), and then proceed with the actual buf array lookup.
More thought needed.
from moss.
From more discussion with @hisundar today, some thoughts are (#1) to split a big segment into multiple smaller segments, which might have a new segment kind (e.g., "a-compressed"). These need to be uncompressed whenever a segment is read.
Another idea (#2) is that each SegmentLoc might also track its minKey and maxKey, and track whether they are non-overlapping with their neighbors, allowing for potential binary-search amongst SegmentLocs in order to speed up lookups (don't have to examine a SegmentLoc if it doesn't have the key you're looking for). This brings LevelDB inspired ideas into the picture, where levels in LevelDB > level 0 may be non-overlapping in keyspace.
from moss.
Related Issues (20)
- leveled compaction only considers top/root collection
- Add IncrementalPersistence method to Segment interface & decouple from compaction HOT 1
- Q on write amplification HOT 2
- TestOpenStoreCollection fail, store_test.go:1035 HOT 2
- occasionally sync during compaction HOT 3
- Build failures on non-amd64 architectures HOT 13
- potential garbage of kvs pair during compactWriter.Mutate()
- status of moss / production ready? HOT 1
- race to complete data write HOT 19
- optimization - same length keys HOT 1
- Q: Transaction performance HOT 2
- Buckets? HOT 2
- Better API? HOT 4
- Moss is using undefined type from ghistogram HOT 4
- Are calls atomic? HOT 1
- Differences from Nitro? Performance/design?
- Feature Request: Add ability to serialize and deserialize Batches and Collections HOT 6
- Moss panic with runaway disk usage HOT 4
- StoreOptions parameter needed for both OpenStore and OpenCollection?
- graphplot deprecated?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from moss.