juju / blobstore

This package provides a Mongo GridFS-backed blob storage engine.

Juju is an open source application orchestration engine that enables any application operation (deployment, integration, lifecycle management) on any infrastructure (Kubernetes or otherwise) at any scale (development or production) in the same easy way (typically, one line of code), through special operators called ‘charms’.

👉	Juju	Learn how to quickly deploy, integrate, and manage charms on any cloud with Juju. It's as simple as juju deploy foo, juju integrate foo bar, ..., on any cloud.
	Charmhub	Sample our existing charms on Charmhub. A charm can be a cluster (OpenStack, Kubernetes), a data platform (PostgreSQL, MongoDB, etc.), an observability stack (Canonical Observability Stack), an MLOps solution (Kubeflow), and so much more.
	Charm SDK	Write your own charm! Juju is written in Go, but our SDK supports easy charm development in Python.

Give it a try!

Let's use Juju to deploy, configure, and integrate some Kubernetes charms:

Set up

You will need a cloud and Juju. The quickest way is to use a Multipass VM launched with the charm-dev blueprint.

Install Multipass: Linux | macOS | Windows. On Linux:

sudo snap install multipass

Use Multipass to launch an Ubuntu VM with the charm-dev blueprint:

multipass launch --cpus 4 --memory 8G --disk 30G --name tutorial-vm charm-dev 

Open a shell into the VM:

multipass shell tutorial-vm

Verify that you have Juju and two localhost clouds:

juju clouds

Bootstrap a Juju controller into the MicroK8s cloud:

juju bootstrap microk8s tutorial-controller

Add a workspace, or 'model':

juju add-model tutorial-model

Deploy, configure, and integrate a few things

Deploy Mattermost:

juju deploy mattermost-k8s

See more: Charmhub | mattermost-k8s

Deploy PostgreSQL:

juju deploy postgresql-k8s --channel 14/stable --trust

See more: Charmhub | postgresql-k8s

Enable security in your PostgreSQL deployment:

juju deploy tls-certificates-operator
juju config tls-certificates-operator generate-self-signed-certificates="true" ca-common-name="Test CA"
juju integrate postgresql-k8s tls-certificates-operator

Integrate Mattermost with PostgreSQL:

juju integrate mattermost-k8s postgresql-k8s:db

Watch your deployment come to life:

juju status --watch 1s

(Press Ctrl-C to quit. Drop the --watch 1s flag to get the status statically. Use the --relations flag to view more information about your integrations.)

Test your deployment

When everything is in active or idle status, note the IP address and port of Mattermost and pass them to curl:

curl <IP address>:<port>/api/v4/system/ping

You should see the output below:

{"AndroidLatestVersion":"","AndroidMinVersion":"","IosLatestVersion":"","IosMinVersion":"","status":"OK"}

Congratulations!

You now have a Kubernetes deployment consisting of a Mattermost backed by PosgreSQL with TLS-encrypted traffic!

Clean up

Delete your Multipass VM:

multipass delete --purge tutorial-vm

Uninstall Multipass: Linux | macOS | Windows. On Linux:

snap remove multipass

Next steps

Learn more

• Read our user docs
• Read our developer docs

Chat with us

Read our Code of conduct and:

• Join our chat: Matrix
• Join our forum: Discourse

File an issue

• Report a Juju bug on Launchpad
• Raise a general https://juju.is/docs documentation issue on Github | juju/docs

Make your mark

• Read our documentation contributor guidelines and help improve a doc
• Read our codebase contributor guidelines and help improve the codebase

blobstore's Issues

ResourceStorage.Get should return ReadSeekCloser

ResourceStorage.Get should return ReadSeekCloser
Then it's part of the contract of the method. calculateExpectedHashes should not need to do a layer-breaking dynamic type conversion.

uploading with challenge-response is easy to get wrong

There are quite a few cases to get right - the blob might be currently being uploaded by another client; it might fail after the response is generated because the target might then have been deleted. It would make for a much simpler client interface if the client did not need to concern itself with these errors.

blob store can create unreferenced blobs

If the server crashes while a resource is being removed, a blob can be left permanently unreferenced (see managedStorage.RemoveForEnvironment). The same applies when uploading new content for an existing path (see mangedResource.putResourceReference)

PutForEnvironmentRequest always deletes request

The first thing that PutForEnvironmentRequest does is to delete the
request from the request queue. This means that any malicious client
can remove requests for other clients by inventing request ids,
thus disrupting other uploads.

resourcecatalog creates documents with $inc and $gt as keys.

resourcecatalog creates documents with $inc and $gt as keys.

blobstore/resourcecatalog.go

Line 182 in 0605600

Update: bson.D{{"$inc", bson.D{{"refcount", 1}}}},

and Line 218 and 219 also.

We have all documents in resourcecatalog with "a" : { "refcount" : { "$gt" : 1 } }, "u" : { "$inc" : {"refcount" : -1 } in them.

Mongodb 2.4 (this or older was current when resourcecatalog was written) did not validate document keys. New mongodb does this validation.

https://docs.mongodb.com/manual/reference/limits/#Restrictions-on-Field-Names

Upgrading from 2.4 to 2.6 fails for resourcecatalog collection with an error for every document in the collection:

Document Error: document is no longer valid in 2.6 because DollarPrefixedFieldName: $gt is not valid for storage.: {  "_id" : ObjectId("56ebaaa75a87741596b0a8cb"),  "n" : "ca0080a3",  "o" : [ { "c" : "storedResources", "d" : "1b2ad7cc57f3b2d4530026f43adf318c9fd11ec004cf9e54cd7a4a8ee6cc2508af4fcbf739
e466fb911c4559a0a969ba", "a" : { "refcount" : { "$gt" : 1 } }, "u" : { "$inc" : { "refcount" : -1 } } } ],  "r" : [ NumberLong(38) ],  "s" : 6 }

preprocessUpload panics on error

The variable f is declared as a return parameter, which means it's
assigned to before the defer runs, which means that if err != nil,
f is always nil, so f.Remove() panics.

This happens, for example, when the data length does not match
the expected length.

Session is not restored when mongo is restarted

We had outages on jujucharms.com due to mongo restarting on config_change hooks.

When mongo restarts, the sessions in gridfs.go are not restored and connection is lost. Sessions need to be copied.

Way to check:

• start mongo, start service, make a query - works.
• kill mongo, make a query - fails.
• restart mongo, make a query - still fails.

ProofOfAccessResponse uses unexported type putResponse

Although technically possible, it's not very nice for
a public API to have functions that take unexported types.

partially uploaded blobs can cause problems

"Resource not available because upload is not yet complete" is not a useful error for a client to get - it means that any client has to be prepared to poll until the blob has actually been uploaded.

Also, a malicious client can potentially deny access to a chosen blob for an arbitrary amount of time by uploading it slowly.

If the server crashes while a blob is being uploaded, the blob will be permanently unavailable.

managedStorage.PutForEnvironment does not close temp file

PutForEnvironment removes the temporary file but does not close it.

PutForEnvironmentRequest will not work across multiple servers

Outstanding requests are stored in memory. This means that if the charm store is split across several servers and the ProofOfAccessResponse call arrives on a different server from the server that ran the PutForEnvironmentRequest, it will not find the request, so it will fail.