joshaber / freezer

Database as a value.

Freezer

Freezer is inspired by / ripped off from Rich Hickey's talk, The Database as a Value and Datomic.

Freezer is an immutable value store. All changes to the store create a new database, as opposed to changing the database in place. Once you have pulled a database out of the store, it never changes. It is a fixed view of the store at that point in time.

Freezer is made up of a few basic components:

• FRZStore is the top-level thing. The store is where you go to get the current database or get a transactor to perform changes.
• FRZDatabase is how you get data out of the store. It is immutable. It never changes in place.
• FRZTransactor is responsible for adding or removing values from the store.

IDs

IDs are used to group key-value pairs into a single entity. They can be any arbitrary string.

Keys

Freezer does not store objects. Instead, it stores an arbitrary collection of key-value pairs, grouped by an ID.

A collection of keys, grouped by ID, may be transformed into, for example, a Mantle model object. But note that model object is simply a view on the data. It is not the thing itself.

Changes

Stores provide a RACSignal of changes which are applied to the store. This signal can be filtered, throttled, etc. as needed to find out about the changes your app cares about.

Thread-safety

Freezer is completely thread-safe. Databases may be read from in any thread and FRZTransactor may add or remove values in any thread.

If the database is immutable... does that mean the store grows without bound?

Yuuuuuuuuuup. But you can call -[FRZTransactor trim:] to trim the store down to the latest database.

Should I use this?

Nooooooooope. Or at least not yet. Freezer's still in a constant state of flux.

freezer's Issues

SQLite VACUUM

We should run VACUUM. Probably as part of trimming.

First-class collection support

Probably easiest to leverage -keysWithAttribute and use some attribute as a belongs-to-this-collection tag.

Get the database at another point in time?

It'd be useful/cool to be able to say, "give me the database from yesterday", etc.

Reify transactions

Transactions exist currently in the database, but we don't expose them to users. It could be useful:

• Timestamp for changes.
• List of changes.

Related to #3.

Implement attributes as columns on a single table?

Currently each attribute is its own table. We could instead implement each attribute as a column on one table. This could help address #5.

The downside is that we'd probably end up with a super wide table. I'm not sure how Sqlite feels about that.

Collections are expensive

Currently, getting a collection of size n requires (at least) n * 2 selects.

That's super gross.

Need any indexes?

We currently don't use any indexes. It'd probably make sense to index on key. Maybe attribute too?

Dates aren't returned as dates

We have a date attribute type, but the object we get out isn't a date. 😢

Move back to a single table

In light of #5, #12, #19, I think it'd make sense to move back to using a single table.

👍

• Simper, more consistent implementation.
• Faster key-based reads.

👎

• The value column will just be a blob, making it harder for us to leverage SQLite magic.
  • We could maybe mitigate that with custom functions which unpack the blob?
• Less space-efficient, presumably.

Cache attribute information

Reads and writes require that we look up attribute information (type, cardinality, etc.). We should cache that.

We'd get this for free if we did #8.

Move to a different backend?

Sqlite's awesome, LevelDB might be more awesome.

👍

• Faster than Sqlite
• Snapshots are a non-persistent version of our immutable databases.
• More thread-safety out of the box.
• Sorted keys could allow for a bit more flexibility.

👎

• Outside dependency.
• Snapshots can't be persisted.
• C++

Speed up key-based reads

Currently key-based reads require walking over each of attribute tables and checking if they have an entry for the key. This means it's gonna slow down as you add more attributes.

A couple thoughts on speeding it up:

• Partition keys so that we can known based on the they that it could only have certain attributes.
• Keep an index of the attributes which a key has.

Sqlite database grows without bound

Currently the sqlite database just keeps adding things. That's less than ideal. We should have some way of trimming it.

Some things:

• We could trim down to the most recent level attribute-value for all keys.
• But it's tricky to know that someone's not holding on to an old database.
• How does this interact with #3?

Rename FRZDatabase to FRZSnapshot

From @mdiep.

It seems like a good idea to make it clearer what's happening.

Pay property tax

Pthread-specific values are being leaked

Just laziness on my part, but we need to fix it.

API to remove attributes

Add a way for users to remove an attribute.

More sophisticated querying?

Should Freezer provide more sophisticated querying?

• Most query APIs are heavily stringly typed.
• Could we use Sqlite functions to get more typing?

Re-think terminology of key/attribute/value

From @mdiep.

I think the dissonance for me is that it feels like I'm mutating a dictionary, but the names don't convey that

I agree. It's tricky because dictionaries are key -> value, but Freezer's more like key -> (attribute, value).

Inserting is O(n) (lolcry)

Currently on both branches, inserting is O(n), n = number of objects in the previous "database." This is because they create a new list of objects for each insert. That's... less than ideal.

In-process caching at the database level?

Since FRZDatabases are immutable, it'd be super easy to cache lookups in process. Should we?