Opening encrypted databases with `cblite` about couchbase-mobile-tools

[snej]

Yeah, we're forbidden to release the SQLite Encryption Extension source code. A few bits of our EE functionality are closed-source just to act as a roadblock to competitors who might want to copy our inhumanly-advanced technology 👽😜

[snej]

It shouldn't be a problem unless the file format changes incompatibly, which doesn't happen much.

In the future I'm hoping we can make the tool official, so there will be updated binary builds available for all supported platforms.

[snej]

if what you're given really is someone's password, you should hash it in some way.

A password has to be hashed, because encryption algorithms use fixed-size keys. PBKDF2 actually runs thousands of rounds of hashing, to slow it down so that brute-force cracking by testing lots of passwords will take a long time.

the keys we've been using are 32 bytes, although switching to 64 wouldn't be the end of the world

Our keys are 32 bytes. That's 64 digits.

[snej]

I've pinged @borrrden to build it and send me the EXE. I'll upload it to the Releases tab.

[ngray-jnj]

Excellent! Just opened an encrypted database with it! Thank you for your help!

[borrrden]

We will be releasing another one when we release 2.8 of Couchbase Lite, which is scheduled for mid next month.

[snej]

You can’t build the Enterprise Edition yourself. That’s sort of the point; it’s the thing Couchbase charges money for. If you want to use EE features, contact our sales team to get a license.

Some parts of CBL-EE are closed-source. In particular, database encryption is handled by the SQLite Encryption Extension, which is likewise a commercial product, that we’ve licensed from the SQLite developers. It’s not available without a license from them.

[ngray-jnj]

Yes, I understand that enterprise features require a license/money. But it was unclear to me until right now that the enterprise code was closed-source, especially because the enterprise nuget packages are publicly available without prior purchase of a license.

[ngray-jnj]

That and Couchbase support seemed unaware of the permissions for the couchbase-lite-ee repository, as they directed me to try to find it without pointing me toward licensing.

[ngray-jnj]

Gotcha, so can we get to the SQLite source code with a Couchbase license? I assume we need that to build cblite in addition to the Couchbase source.

[snej]

There is no way to obtain the SQLite Encryption Extension source code without getting your own license from SQLite.

Sorry, I just now realized that your basic question is how to use the cblite tool with an encrypted database. The answer is that we should provide a binary of cblite built with EE. I'll try to make that happen.

[snej]

What platform are you on? I can do a Mac build myself; others I'd need to ask @borrrden.

[ngray-jnj]

No worries. I'm on Windows, and everyone else on the team who needs cblite will be using Windows as well. If you provide me with a binary, how can we make sure to have the latest EE build for cblite in the future?

[ngray-jnj]

Okay great. Do you also have instructions for using the EE version that details the differences in the interface? i.e. will I need extra guidance to figure out how to pass an encryption key to cblite to inspect the encrypted database?

[snej]

Good point; I'll document those when we upload a binary. IIRC there's a --password flag to give the password/key.

[ngray-jnj]

FYI, we've been using the Couchbase Lite EncryptionKey constructor that takes a byte array in order to encrypt and decrypt the database. Does the --password parameter get decoded into a byte array using a particular encoding scheme? If we know that we can convert our encryption keys to strings and use them with cblite.

[snej]

We use the PBKDF2 algorithm to convert a password to a key. That operation isn't reversible.

Currently the tool will try to open the db, then if it detects that it's [probably] encrypted, it will prompt for a password on stdin. I can improve that to support a raw key; the easiest way would be to detect if you've entered 64 hex digits and parse that as a raw key not a password.

[ngray-jnj]

Oh right, if what you're given really is someone's password, you should hash it in some way.

That enhancement would be great for us; the tool won't really work for our use case if it can't accept a raw key. For better or worse, our design hinges on supplying the database with our own key. However, the keys we've been using are 32 bytes, although switching to 64 wouldn't be the end of the world for us by any means. Would the update you're proposing be flexible enough to detect both 64 and 32 hex digits? Because the Couchbase Lite databases certainly allow a range of key sizes, though I'm not certain what the upper and lower limits are.

[ngray-jnj]

Right, forgot how hex works. 😢

I see you already made the relevant commit. How do I go about getting the binary from you?

[snej]

Someone else here who has Windows and MSVC will have to build it, most likely @borrrden.

[ngray-jnj]

Great, just keep me posted.

[ngray-jnj]

Any updates on the build for this?

[snej]

Come and get it!

[ngray-jnj]

Hello cblite team, would it be possible to get another EE build with the latest features, in particular the lovely "info" command?

[ngray-jnj]

Is there another way to view the contents of an index, even if that's through the SDK?

[snej]

No. The mechanism for inspecting a SQLite index is kind of ugly, and the SQLite docs warn that it shouldn't be relied on in production, so there is deliberately no public CBL API for it. The cblite tool is using a private API that was put there for its use only.

Until we release another EE build, you could decrypt a copy of your database, build your own CE cblite tool, and work with the decrypted db.

[ngray-jnj]

Ah, that's unfortunate, but obviously not your fault. The database I happen to be inspecting isn't one of our encrypted databases, so I can actually just build the CE cblite as you say and get that to do what I need for now.

Thanks!