Comments (10)
On the other hand, it starts to look like a layering violation - page freelist is a property of a pager; I'll experiment with the best place to put such a virtual interface a little more and come back with a definitive answer. Perhaps a pager needs a pVfs table for virtual functions related to I/O, and a separate virtual table for an allocation scheme.
from libsql.
I wondered if it should be abstracted away to another virtual table, since it doesn't technically need to touch disk, like other io_methods, but managing free pages is tightly bounded to disk operations anyway, so sqlite3_io_methods
indeed sounds like the best place.
from libsql.
The actual implementation is even deeper, in BTree:
https://github.com/libsql/libsql/blob/aafe1d5457936a05bca5d312e2fd25f32afeb20b/src/btree.c#L6244
https://github.com/libsql/libsql/blob/aafe1d5457936a05bca5d312e2fd25f32afeb20b/src/btree.c#L6566
@losfair to clarify, you're interested in being able to override the existing hardcoded freelist management, which uses offsets 32 and 36 of each page to maintain a freelist, and instead be able to implement your own scheme, right? It's important to note that after it's done, the database will most likely only be readable if you later run it with the same exact page allocation/deallocation implementation set up. Right now the freelist is persisted on disk, so that the database is able to check which pages are free during bootstrap.
from libsql.
@psarna Yes I would like to be able to override the builtin freelist-based free page management - the now-unused word at offset 32/36 can be filled with zero then?
It's important to note that after it's done, the database will most likely only be readable if you later run it with the same exact page allocation/deallocation implementation set up.
Wouldn't this only break writes but not reads under the default configuration?
from libsql.
Alright, then I'll take a stab at providing a way for virtualizing the freelist management and send it for peer review as a pull request once done.
It's important to note that after it's done, the database will most likely only be readable if you later run it with the same exact page allocation/deallocation implementation set up.
Wouldn't this only break writes but not reads under the default configuration?
Well, depends on what the implementation does - in general case, whatever's stored at offsets 32 and 36 can only be properly interpreted by the same (or backward compatible) implementation. But I think it's safe to assume that users intending to override freelist management never need to suddenly change its implementation on an existing file.
By the way, do you have a specific implementation in mind, and can you share any details on it? With a custom implementation, the information about which page is free needs to be persisted somewhere anyway - either at one of these offsets (32,36), or in some external persistent storage, otherwise the database wouldn't be able to figure out which pages are free when bootstrapping.
from libsql.
self-note: the implementation would also need to be very careful to override various sanity checks and assumptions, like this one:
https://github.com/libsql/libsql/blob/aafe1d5457936a05bca5d312e2fd25f32afeb20b/src/btree.c#L10731
after a custom implementation is allowed, any assumption about data at offset 32 or 36 may not hold true anymore, if a non-default page allocation policy is used.
from libsql.
By the way, do you have a specific implementation in mind, and can you share any details on it?
The implementation is mvSQLite. With this change page allocation can be further pushed down to FoundationDB:
- IDs of freed pages are inserted to a "free set" implemented as a KV prefix subspace.
- IDs of newly allocated pages are either selected randomly from the free set, or assigned a previously unused random ID (
new_id > current_max_page_id
).
Offset 32/36 do not need to be reused - the free set is kept internally as database metadata.
from libsql.
Got it, makes perfect sense. I'll continue working on it and ping you once I have something workable.
from libsql.
@losfair just to make sure to bring this to your attention, there's a PR open implementing a first RFC for this, would love to get your eyes on it!
from libsql.
Thanks for implementing this so quickly! Will try it next week.
from libsql.
Related Issues (20)
- ALTER COLUMN breaks if it contains a new line at the end
- `.dump` fails if the request is sent to replica HOT 1
- Support for Accessing SQLite DB Stored in Turso from Java HOT 1
- Extension loading support in libSQL server
- sqlite "ConstraintViolation" reported as a "PROXY_ERROR" HOT 4
- Rows written with `INSERT OR IGNORE` still read/writing HOT 1
- Setting `legacy_alter_table` works in shell but not in sqld HOT 3
- Error while creating index for vector database table HOT 2
- Optimize IFNULL with indexes
- Use `WAL` as default everywhere HOT 5
- Either support or reject BEGIN READONLY in hrana on the server
- Unable to run on `oven/bun:distroless` docker image HOT 4
- Feature Request: Expose `sync_interval` parameter in libsql's C bindings
- Import fails when there are multiline statements in the dump HOT 1
- Bug with schema migration when tasks status do not reflect the job status
- Group tokens with attach permissions lose broader read/write permissions
- Switch to using `gosu` in `docker-entrypoint.sh`? HOT 1
- Schema migrations fail when run on a client connected to a replica HOT 3
- Sqlean version out-of-date in Turso HOT 5
- ERROR libsql::replication: replicator sync | database disk image is malformed HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libsql.