inveniosoftware / cookiecutter-invenio-datamodel

Cookiecutter template for an Invenio data model.

Cookiecutter-Invenio-Datamodel

This Cookiecutter template is designed to help you to bootstrap the data model for an Invenio service.

Quickstart

Install the latest Cookiecutter if you haven't installed it yet:

pip install -U cookiecutter

Generate your Invenio datamodel:

cookiecutter https://github.com/inveniosoftware/cookiecutter-invenio-datamodel.git

Features

• Python package: Python package for your service.
• Boilerplate files: README including important badges, AUTHORS and CHANGES files.
• License: MIT file and headers.
• Installation: Installation script written as setup.py and a requirements calculator for different levels (min, pypi).
• Tests: Testing setup using pytest.
• Documentation: Documentation generator using Sphinx. Also includes all files required for Read the Docs. Mocking module to simulate not-installed requirements for faster documentation building.
• Continuous integration: Support for Travis which tests all requirement levels and adds coverage tests using Coveralls.
• Your toolchain: Ignores a decent set of files when working with Git and Docker. Gets your editor to adapt project guidelines by providing an EditorConfig file.

Configuration

To generate correct files, please provide the following input to Cookiecutter:

project_name	Full project name, might contain spaces.
project_shortname	Project shortname, no spaces allowed, use - as a separator.
package_name	Package/Module name for Python, must follow PEP 0008.
github_repo	GitHub repository of the project in form of USER/REPO, not the full GitHub URL.
description	A short description of the functionality of the module, its length should not extend one line.
author_name	The name of the primary author of the project, not necessarily the same as the copyright holder.
author_email	E-Mail address of the primary author.
year	Current year.
copyright_holder	Name of the person or organization who acts as the copyright holder of this project.
elasticsearch_version	The version of ElasticSearch you are planning to use. Versions 5.x (v5) and 6.x (v6) supported at the moment.
extension_class	Name of the class that will be exported as setuptools entrypoint and loaded by invenio main app.
config_prefix	Prefix for the configuration keys that the main app will use for this extension.

Further documentation is available on https://cookiecutter-invenio-datamodel.readthedocs.io/

cookiecutter-invenio-datamodel's Issues

python 3.6 support

When you run the bootstrap script with python 3.6 you get

...
gyp ERR! configure error
gyp ERR! stack Error: Python executable "/Users/lnielsen/envs/my-site/bin/python" is v3.6.4, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
...\

The problem have been reported before, but not solved as far as I know. Any chance we could get it fixed, so that instructions works for Python 3.6 as well as 3.5

permissions factory allow_all is not working properly

The permissions are not working properly for invenio-records-rest when following the documentation

requirements.txt instructions update

requirements.txt should not include test dependencies, i.e. pip install -e .[all] should be something like pip install -e . instead.

tests: test the generated .travis.yml for the data model

The generated .travis.yml for a new project needs to be tested and we need to make sure that the tests pass.

mappings: incorrect PID field type

The PID name field in the ES mapping should be of type keyword.

On a separate note, I guess the bigger issue that a lot of people might come across, is that their marshmallow schema or JSONSchema might change, but they'll forget to update the mapping (or like in this case, they'll have an incorrect mapping). It would be nice to have something which cross-validates the Marshmallow/JSON/ES schemas...

schemas: harmonise loader/serializer schemas

We now have schemas in loaders/schemas/ and serializers/schemas.

Centralize this into a top-level package so that we have:

loaders
serializers
schemas

docs: INSTALL.rst needs to be updated

The INSTALL.rst needs to be rewritten as it is just a copy/paste from the instance.

marshmallow: incorrect field types

• The keywords field in the json.py marshmallow schema, should be fields.List(SanitizedUnicode()) instead (since it's not a list of objects).
• Probably some of the fields.Str() fields should be SanitizedUnicode instead

tests: add basic tests using pytest-invenio to generated project

Currently there's no tests in the generated project - we need a least one test which covers the basic use case: 1) create a record 2) retreive a record via the API.

We should aim for 100% test coverage on the generated project.

bootstrap script should use requirements.txt

The bootstrap script is current installing everything with pip install -e .. Instead it should probably be

pip install -r requirements.txt
pip install -e .

Document the most important files in the data model

Simplify serialisers to avoid copy/paste code

E.g. all fields should be defined in a common invenio package

Add tests similar to cookiecutter-invenio-module

loaders/serializers: "PersistendIdentifier" field correctly handling "recid"-like fields

skeleton: the generated skeleton contains misc folder

Add post project generation hook that removes the misc folder:

https://github.com/inveniosoftware/cookiecutter-invenio-instance/blob/master/hooks/post_gen_project.sh

loaders: move marshmallow_loader and friends to invenio-records-rest

Move

<datamodel>/loaders/errors.py
<datamodel>/loaders/json.py

to

Invenio-Records-REST:

invenio_records_rest/loaders/marshmallow.py

config: add REST API search facets

RECORDS_REST_FACETS, RECORDS_REST_SORT_OPTIONS, RECORDS_REST_DEFAULT_SORT and possibly other configuration variables that apply on the datamodel level should be included in config.py and added via an update in init_config.

This might require some changes in invenio-records-rest as well (investigation needed)

tests: fix failing travis ci build

Add MIT license to all generated files

Add MIT license headers to all generated files + MIT license file (see cookiecutter-invenio-instance example)

Fix requirements.txt files

Pinned and devel should be removed TBC

Add missing project files

Similar to inveniosoftware/cookiecutter-invenio-module we need:

.editorconfig
.gitignore
.travis.yml
AUTHORS.rst
LICENSE
README.rst
cookiecutter.json
docs
hooks
run-tests.sh

installation: update to invenio-records-rest v1.1.0

It might be better to do so after addressing #19 and #21

config: auto install API endpoints

Something like:

# ext.py
def init_config(self, app):
    for k in dir(config):
        if k.startswith('<CFGPREFIX>_'):
            app.config[k] = getattr(config, k)
        elif k == 'RECORDS_REST_ENDPOINTS' \
                    and app.config.get('<CFGPREFIX>_ENDPOINTS_ENABLED', True):
                app.config['RECORDS_REST_ENDPOINTS'].update(getattr(config, k))

# config.py
<CFGPREFIX>_ENDPOINTS_ENABLED

RECORDS_REST_ENDPOINTS = {
    .... # note: name changed (no prefix)
}

Probably the same should happen with Records-UI

This way 1) by default we add the data models endpoints to the configuration and 2) a user can switch them off if they like to change something.