ipazc / mldatahub Goto Github PK
View Code? Open in Web Editor NEWMachine Learning data HUB for storing datasets.
License: GNU General Public License v3.0
Machine Learning data HUB for storing datasets.
License: GNU General Public License v3.0
The page-size in the backend should behave like a limit for the page size. It should be able for the client to specify a smaller page size.
Tags for datasets or elements currently do not allow complex objects as components, it only allows strings. Perhaps allowing more complex objects would be useful for the usage of mongo engine search options.
This is the error retrieved by dhub when an invalid option is sent:
Exception: Failed to communicate with backend: {"message": "Internal Server Error"}
This kind of errors should be masked.
It would be a good idea to add a cache layer in the storage and save everything in chunks of files of certain size. The storage should not be synchronized, it should be stored in background. To be multiprocess compliant, it should store the file status in the db.
The dataset size global value is not reliable when its elements has been uploaded concurrently.
When the parent of a forked element is changed or the forked element itself, all the forked elements change their ID.
This is a problem since the client expects the elements to keep their ID. This error can be replicated this way:
>>> from dhub.datasets import Datasets
>>> datasets = Datasets()
>>> dataset = datasets.add_dataset("example")
>>> dataset.add_element(title="This is element1", content=b"none")
>>> fork = dataset.fork("example2")
>>> forked_element = fork[0]
>>> print(forked_element.get_content())
b"none"
>>> forked_element.set_title("changed")
>>> print(forked_element.get_content()
KeyError exception raised.
OS: Ubuntu 14.04 x86_64
Interpreter: Python3.4
DHub version: 0.0.14
Error codes are not accompanied by a crafted message. It should be the most descriptive as possible to help in the debug of possible errors.
There is a lack of unit tests for the API entries.
It would be a good idea to have an option to order the dataset's elements.
The options when retrieving elements should be translated from a custom language.
This will decouple the options from the backend mongo, and also give some security to the request.
Just to allow to have a reference for the dataset.
There is a lack in unit tests for checking that a token is limited to a set of datasets or that dataset has a limit in the number of items.
When the dataset is getting stored, there is a chance for it to garbage collect files before they are assigned to a dataset elements. This happens because the operation is not atomic: first it is created the file in the database and, after that, it is referenced in the dataset element.
When the unittests are run, the production DB is purged. Unit tests have their own DB in mongo to test, and it must be ensured that they use this instead.
The modified date is not currently updated. On any modification it should be automatically updated.
Currently the input arguments are not restricted in size. It should perform some size checks.
There is no need to copy the metadata of all the parent's elements. Instead, it should only create new metadata for the element that is edited or deleted.
Currently, the clear command is forwarding each key to delete to the server. When the dataset is huge, the process is so slow and requires multiple requests, because the keys are retrieved in chunks of PageSize length, and the deletion is also controlled by the PageSize.
When updating the same object twice in small amounts of time, the first object changes are not reflected in the second object. This needs a layer of synchronization.
The size of the elements can be stored in the database also, so it can be known the size of a dataset.
The pagination for elements in Mongo might be slow when using skip()
.
An interesting way of solving this problem is referenced here.
Perhaps letting the user sending the last element's id from the page rather than the page number (in order to get the next one) would help in speeding up the results.
The URL prefixes for the dataset seem to be leaked when multiple backends are used at the same time.
This might happen when the dataset is created in one backend but the removal is done by a second backend, leaking the url prefix in the first backend.
When requesting the dataset list, the time grows with the number of elements.
It should not affect to the retrieval time.
DatasetElementFactory(token, dataset).edit_elements(crafted_request)
File "/usr/local/lib/python3.5/dist-packages/mldatahub-0.0.1-py3.5.egg/mldatahub/factory/dataset_element_factory.py", line 224, in edit_elements
files_refs = {element_id: file for element_id, file in zip(elements_ids, self.storage.put_files_contents(elements_content))}
File "/usr/local/lib/python3.5/dist-packages/mldatahub-0.0.1-py3.5.egg/mldatahub/storage/remote/mongo_storage.py", line 65, in put_files_contents
raise Exception("File size limit of {} Bytes exceeded".format(FILE_SIZE_LIMIT))
Exception: File size limit of 16777216 Bytes exceeded
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.