ipazc / mldatahub Goto Github PK

The page-size in the backend should behave like a limit for the page size. It should be able for the client to specify a smaller page size.

Tags for datasets or elements currently do not allow complex objects as components, it only allows strings. Perhaps allowing more complex objects would be useful for the usage of mongo engine search options.

This is the error retrieved by dhub when an invalid option is sent:

Exception: Failed to communicate with backend: {"message": "Internal Server Error"}

This kind of errors should be masked.

It would be a good idea to add a cache layer in the storage and save everything in chunks of files of certain size. The storage should not be synchronized, it should be stored in background. To be multiprocess compliant, it should store the file status in the db.

The dataset size global value is not reliable when its elements has been uploaded concurrently.

When the parent of a forked element is changed or the forked element itself, all the forked elements change their ID.
This is a problem since the client expects the elements to keep their ID. This error can be replicated this way:

>>> from dhub.datasets import Datasets
>>> datasets = Datasets()
>>> dataset = datasets.add_dataset("example")
>>> dataset.add_element(title="This is element1", content=b"none")
>>> fork = dataset.fork("example2")
>>> forked_element = fork[0]
>>> print(forked_element.get_content())
b"none"
>>> forked_element.set_title("changed")
>>> print(forked_element.get_content()
KeyError exception raised.

OS: Ubuntu 14.04 x86_64
Interpreter: Python3.4
DHub version: 0.0.14

Error codes are not accompanied by a crafted message. It should be the most descriptive as possible to help in the debug of possible errors.

There is a lack of unit tests for the API entries.

It would be a good idea to have an option to order the dataset's elements.

The options when retrieving elements should be translated from a custom language.
This will decouple the options from the backend mongo, and also give some security to the request.

Just to allow to have a reference for the dataset.

There is a lack in unit tests for checking that a token is limited to a set of datasets or that dataset has a limit in the number of items.

When the dataset is getting stored, there is a chance for it to garbage collect files before they are assigned to a dataset elements. This happens because the operation is not atomic: first it is created the file in the database and, after that, it is referenced in the dataset element.

When the unittests are run, the production DB is purged. Unit tests have their own DB in mongo to test, and it must be ensured that they use this instead.

The modified date is not currently updated. On any modification it should be automatically updated.

Currently the input arguments are not restricted in size. It should perform some size checks.

There is no need to copy the metadata of all the parent's elements. Instead, it should only create new metadata for the element that is edited or deleted.

Currently, the clear command is forwarding each key to delete to the server. When the dataset is huge, the process is so slow and requires multiple requests, because the keys are retrieved in chunks of PageSize length, and the deletion is also controlled by the PageSize.

When updating the same object twice in small amounts of time, the first object changes are not reflected in the second object. This needs a layer of synchronization.

The size of the elements can be stored in the database also, so it can be known the size of a dataset.

The pagination for elements in Mongo might be slow when using skip().
An interesting way of solving this problem is referenced here.

Perhaps letting the user sending the last element's id from the page rather than the page number (in order to get the next one) would help in speeding up the results.

The URL prefixes for the dataset seem to be leaked when multiple backends are used at the same time.
This might happen when the dataset is created in one backend but the removal is done by a second backend, leaking the url prefix in the first backend.

When requesting the dataset list, the time grows with the number of elements.
It should not affect to the retrieval time.