Prevent duplicates in blob store? about crate

[lukasender]

You could, for example, create a table 'file_index' which keeps a reference to a file for each user.
E.g.

CREATE TABLE file_index (
    user_name string,
    file string
);

If a user then deletes a file, you could only delete the reference which points to that file. The file would still be there for other users.

Could such a solution be of help?

[ddorian]

That is doable, but it would suck. Have to do a count(*) before every insert/delete.

[dobe]

hi @ddorian

for inserts:
if you use the username and the digest of the file as primary key, there is no check required anymore on insert to prevent duplicates.

the "cleanup" of unused digests could be done asynchronously by a periodic task by selecting all used digests from your files table and comparing them to the existing digests. to prevent race conditions you could just delete files older than a day.

getting all stored digests of a blob table is currently not supported, but is already in progress and should make it into one of the next releases.

one other solution is to create an inode table with filenames like on normal filesystems, where the filename plays the same role as the username described above.

if you have other feature suggestions we would appreciate your feedback

thx, bernd

[ddorian]

So i can put anything i want in the 'digest' field when i write/upload a file ?

How are blobs stored? Is all the file stored in one server (not broken into pieces like riakcs) ?

[dobe]

So i can put anything i want in the 'digest' field when i write/upload a file ?

i assume that you mean BLOB when you say file - so the answer is no, since the digest of a blob needs to be the hex encoded sha1 hash of the content (see also) https://crate.io/docs/current/blob.html . the digest is actually not a field it is the identifier of the blob, when a blob changes the digest changes too. note that there is currently no thing like a filesystem api on top of the crate BLOB support, but we have it on our roadmap.

How are blobs stored? Is all the file stored in one server (not broken into pieces like riakcs) ?

no blobs are not broken into pieces, they are stored as is. as of now the actual files on the filesystem can be verified via its name, since the name of the blob on the filesystem is the hash, this also makes blobs immutable.

[ddorian]

On Mon, May 5, 2014 at 2:52 PM, Bernd Dorn [email protected] wrote:

So i can put anything i want in the 'digest' field when i write/upload a
file ?

i assume that you mean BLOB when you say file - so the answer is no, since
the digest of a blob needs to be the hex encoded sha1 hash of the content
(see also) https://crate.io/docs/current/blob.html . the digest is
actually not a field it is the identifier of the blob, when a blob changes
the digest changes too. note that there is currently no thing like a
filesystem api on top of the crate BLOB support, but we have it on our
roadmap.

Why can't i store a unique-provided-by-the-client key instead of the
digest? Does the server confirm the digest?

How are blobs stored? Is all the file stored in one server (not broken
into pieces like riakcs) ?

no blobs are not broken into pieces, they are stored as is. as of now the
actual files on the filesystem can be verified via its name, since the name
of the blob on the filesystem is the hash, this also makes blobs immutable.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/735#issuecomment-42184206
.

[dobe]

Why can't i store a unique-provided-by-the-client key instead of the
digest? Does the server confirm the digest?

yes, the server validates the digest, also it is used for routing the file to the right shard.