Add blob storage option to store files on s3 about crate

[dobe]

I don't know if you are aware of it, but there is builtin BLOB support in Crate. See https://crate.io/blog/using-crate-data-as-a-blobstore/ for an introduction and the documentation https://crate.io/docs/stable/blob.html

[martinheidegger]

The internal BLOB support is good and useful. I am talking about an option to flip a switch and have crate automatically store the blobs on s3 (or another file storage system).

[dobe]

Ah, so you mean that Crate should act as a kind of proxy to 3rd party storages. I think this use-case is very seldom, so probably we will not put resources on this. We think that in most cases, one will use Crate as the BLOB store because there is no additional setup required. Also local disks are cheaper in most of the cases.

If you really need S3, it would make sense to access it directly, since a Crate proxy will give you no benefit in this case and will always have the same latency as S3.

[martinheidegger]

At a first glance I can see three possible benefits for external storage of blobs:

1. It is possible to use the same code no matter if you use your local test environment or your productive environment on s3.
2. You only need to know the connection string to crate to access all your data (blobs and text content)
3. If offered flexibly it could work as a import/export option (i.e. you could easily import blobs from/to s3)

[martinheidegger]

Amendment: s3 as backup/seed option would also give me confidence that the system really is fail-safe :)

[dobe]

yep, we have an s3 backup option like "copy to" for blobs on our roadmap.

[martinheidegger]

But that is just a backup right? There is another reason I just remembered for a blob storage strategy on s3

• you will never ever run out of disk space.

[weswam]

I am not a code contributor but I did want to briefly chime in on this. I honestly see zero benefit to having Crate store its blob data in S3. I have actually used crate and its built in blob store to completely move all of our in house data off of S3. Crate allowed us to build an in house replacement for S3 and I believe that was one of the original intents of the Crate blob store. It allows you to store your blobs locally in your own storage cluster. The added latency from connecting to Crate and then having it connect to S3 is never going to be beneficial. You would be better served to store your meta data in Crate and then connect directly to S3 within your application than to have Crate act as a proxy to an external data store such as S3. It seems that your main concern is around not having enough disk space to store your data and really that is easily solved buy building your own storage cluster. It most likely is not cost effective to do so on VMs but dedicated hardware is cheap and plentiful once your project is getting to a scale where you need to be seriously concerned about it and honestly once you have that amount of data local disks is way cheaper than S3.

[dobe]

i think @weswam is right, the usecase to use crate as a s3 proxy is very seldom.