inveniosoftware / cookiecutter-invenio-instance Goto Github PK
View Code? Open in Web Editor NEWCookiecutter template for an Invenio instance.
License: MIT License
Cookiecutter template for an Invenio instance.
License: MIT License
During boostrap of a new cookiecuter you get the following warning about NPM packages:
audited 11234 packages in 9.794s
found 3 vulnerabilities (2 low, 1 moderate)
run `npm audit fix` to fix them, or `npm audit` for details
Following the instruction in here, after completing all the prior steps without any error. the step ./scripts/setup
, ends in following error
Cache cleared
Destroying database postgresql+psycopg2://uebibrepo:uebibrepo@localhost/uebibrepo
Creating database postgresql+psycopg2://uebibrepo:uebibrepo@localhost/uebibrepo
Creating all tables!
[####################################] 100%
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Created all tables!
Traceback (most recent call last):
File "/usr/local/bin/invenio", line 11, in <module>
sys.exit(cli())
File "/usr/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 412, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/invenio_search/cli.py", line 34, in inner
cluster_ver = current_search.cluster_version[0]
File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
File "/usr/local/lib/python2.7/dist-packages/invenio_search/ext.py", line 221, in cluster_version
versionstr = self.client.info()['version']['number']
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 241, in info
return self.transport.perform_request('GET', '/', params=params)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 85, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', BadStatusLine("''",))) caused by: ConnectionError(('Connection aborted.', BadStatusLine("''",)))
the comment was executed both virtualenv enabled as of pipenv install -e ./my-datamodel
its output message saying to run pipenv shell
and without virtualenv, both ends up in same above error. Requesting help in relsoving this error.
For the Flask-Debugtoolbar
extension to work, it must also be added the invenio_base.apps
entrypoints
Another problem though is that CSP headers are not playing well with the extension: pallets-eco/flask-debugtoolbar#88
Usage:
$ cookiecutter cookiecutter-invenio-instance/
project_name [My site]:
...
Select default data model:
1 - DataCite
2 - DublinCore
3 - MARC21
4 - Custom (as part of the instance)
5 - No data model
Choose from 1, 2, 3, 4, 5 [1]:
(options 1,2,3 not being available today)
Option 5: Like today
Option 1: Cookiecutter-invenio-datamodel
theme
package (and register package in setup.py)When you run the bootstrap script with python 3.6 you get
...
gyp ERR! configure error
gyp ERR! stack Error: Python executable "/Users/lnielsen/envs/my-site/bin/python" is v3.6.4, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
...\
The problem have been reported before, but not solved as far as I know. Any chance we could get it fixed, so that instructions works for Python 3.6 as well as 3.5
(copied from inveniosoftware/cookiecutter-invenio-datamodel#10)
When starting the docker containers:
docker-compose -f docker-compose.full.yml up
we get the following error:
Step 19/23 : RUN ./scripts/boostrap ${WORKING_DIR}
---> Running in a574736bb0ff
/bin/sh: 1: ./scripts/boostrap: not found
INSTALL.rst
to mention (#111):
pipenv shell
(ask Lars for where to put it)UWSGI
-> uWSGI
.project_site
cookiecutter variable whenever a domain is needed. E.g. in config.py
(#110):
SUPPORT_EMAIL = 'info@{{cookiecutter.project_site}}'
APP_ALLOWED_HOSTS = ['{{cookiecutter.project_site}}', 'localhost', '127.0.0.1']
Stemmed from #43 (comment).
docker-compose.full.yml
is currently exposing port 5000 for the web-ui
container and port 5001 for the web-api
container. This is not necessary since these are only accessible by nginx (the frontend
container) inside the Docker network, so they could both just expose port 5000 and be accessed in the nginx config as web-ui:5000
and web-api:5000
.
Once inveniosoftware/invenio-app#36 has been merged and released, the /ping
view can be removed (and probably replaced with a generic "Hello world" kind of view...)
Define a strategy for how this repository should evolve with new Invenio versions.
Assume e.g. Invenio v3.0.0, 3.0.1, 3.1.0; 3.0.1 is fully compatible with 3.0.0, however, 3.1.0 is not guaranteed to be fully compatible with 3.0.x.
setup.py
requires invenio[...]~=3.0.0
- this will take care of installing 3.0.0 or 3.0.1 (once out). Once v3.1.0 is out, we need to update the requirement to invenio[...]~=3.1.0
, but we should still support cookiecutter for Invenio v3.0.0 (not indefinitely but for a defined amount of time).
This also relates to how people use this cookiecutter template.
E.g. we could create a branch for each minor (i.e. 3.0 -> 3.1) release of Invenio, and ask users to cookiecut that branch. I.e. something like (not sure if below is possible):
$ cookiecutter https://github.com/inveniosoftware/[email protected]
Last but not least, we may benefit from writing into the repository the commit hash or similar of the cookiecutter template used to generate a project (to later allow for diffing etc).
Use cookiecutter to generate a new template, push to github, enable travis and see that the generated .travis.yml works.
docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4 -> docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0
Currently https://github.com/inveniosoftware/cookiecutter-invenio-instance/blob/master/cookiecutter.json specifies v5 as the default because it's listed first.
https://inveniosoftware.org/#getstarted
instead of invenio.readthedocs.io# TODO: Redefine these permissions to cover your auth needs
./scripts/boostrap
npm WARN [email protected] requires a peer of eslint-plugin-standard@>=4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@^2.2.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@1 || ^2 || ^2.1.0-beta || ^2.2.0-rc || ^3 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@^1.0.0 || ^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.
./scripts/bootstrap
audited 11234 packages in 9.794s
found 3 vulnerabilities (2 low, 1 moderate)
run `npm audit fix` to fix them, or `npm audit` for details
WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
adminlte (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/2.354a92ac4634091f738b.js
base (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/3.cb9fb27fbfa249f933bd.js
i18n_app (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/4.5e976e0d6bbd39f439e4.js
search_ui_app (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/5.abc761eadb0ee2b8bdaa.js
search_ui_theme (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
css/search_ui_theme.d63a24f731803c9f2d46.css
js/6.23be2997be0d9fd0c6ca.js
theme-admin (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
css/theme-admin.35fbd8638b957bb7bd67.css
js/8.526bbb47d1c08bcdb882.js
Persistent identifier serialization problems:
/records/
from the search page) my-site/INSTALL.rst should include how to activate a virtualenv pipenv shell
and change UWSGI -> uWSGI
.
It contains: testpaths = docs tests/unit my_site
but should probably be testpaths = docs tests my_site
Please check if any other PR is not already fixing it.
Running the bootstrap script you get the following warnings:
npm WARN [email protected] requires a peer of eslint-plugin-standard@>=4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@^2.2.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@1 || ^2 || ^2.1.0-beta || ^2.2.0-rc || ^3 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of webpack@^1.0.0 || ^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.
The setup script creates two users with hard-coded email addresses. Perhaps it's better to either not create users or to parameterize it in cookiecutter for just a single user?
setup.py
includes an entry point to a translation folder, also babel is required and there exists a .tx
folder.
There's however no translation folder and no instructions for how to run python setup.py compile_catalog
.
Add support for using a different license that MIT or no license at all.
When you run the bootstrap script you get warnings about very big (2.2MB) bundles
WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
adminlte (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/2.354a92ac4634091f738b.js
base (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/3.cb9fb27fbfa249f933bd.js
i18n_app (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/4.5e976e0d6bbd39f439e4.js
search_ui_app (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
js/5.abc761eadb0ee2b8bdaa.js
search_ui_theme (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
css/search_ui_theme.d63a24f731803c9f2d46.css
js/6.23be2997be0d9fd0c6ca.js
theme-admin (2.24 MiB)
js/manifest.971ef611d8fa360ea500.js
css/vendor.cb86bc6cb135e38cbe31.css
js/1.f8179b682b4f1957cc56.js
css/theme-admin.35fbd8638b957bb7bd67.css
js/8.526bbb47d1c08bcdb882.js
The generated INSTALL.rst
should use the bootstrap/setup scripts.
Flow of documentation (needs verification):
virtualenv
./scripts/bootstrap
docker-compose up -d
./scripts/setup
./scripts/server
The use of the boostrap script in the Dockerfile was removed because we need to pass --unsafe
or --user=root
. See #91
Instead probably the boostrap script should be fixed to allow for this. See discussion https://github.com/inveniosoftware/cookiecutter-invenio-instance/pull/91/files#r241034765
Take the update in #124 and make the tests pass. Problem seems to be related to the docker-compose.full.yml
and pipenv not being available (see failed travis build).
In particular:
CORS
: Currently the default settings of the new instance + default settings of Invenio-REST seems to have CORS enabled with wildcard allow origin.SECRET_KEY
: Does it need to be defined in the config? Or just documented that it needs to be set.APP_ALLOWED_HOSTS
: Should we set it (e.g. localhost
) or just document it?Investigate how documentation and the docker setup can be improved for the docker-compose.full.yaml to avoid confusion for newcomers.
See e.g. gitter conversation: https://gitter.im/inveniosoftware/invenio?at=5b520e9e05e1cc3553fcff62
Note: already present in the docker-services.yml
The quickstart guide show the PID in the JSON. However, when I try to run the quickstart guide the ID is not in the output. This impacts
/scripts/setup
currently does not clean redis.
Not sure how to best clean redis since we have several different modules using redis:
Also, the queue should be purged:
invenio index queue init
invenio index queue purge # <- add
We possibly need a setup.py
file in the root folder for documentation build dependencies. At this point the setup.py
file that is available is the one that is used as a template for the generated instance, and is not at the root directory.
The documentation of the cookiecutter
and the documentation of the created instance from cookiecutter are two different things.
A possible workaround could be to build the documentation another way with make
and to skip installation, in the admin console of read the docs in advanced section, on top there is an option Install Project
which we might need to uncheck.
After running the:
docker-compose -f docker-compose.full.yml up
I tried to initialize the DB with:
docker run -it test-invenio-web-ui /bin/bash
test-invenio db init create
but I got the following error:
Creating database postgresql+psycopg2://test-invenio:test-invenio@localhost/test-invenio
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 403, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 532, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 1193, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 1190, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 477, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 410, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.5/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
Here is the output of docker ps
(ports looks ok)):
»»»» docker ps 0|17:06:09
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9f3a8acfc7d test-invenio-web-ui "/bin/bash" Less than a second ago Up 3 minutes focused_dubinsky
b4561bc2bfbd test-invenio-lb "/docker-entrypoint.…" 3 hours ago Up 4 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8080->8080/tcp test-invenio_lb_1
8583a46e4a6c test-invenio-frontend "nginx -g 'daemon of…" 3 hours ago Up 4 minutes 0.0.0.0:32785->80/tcp, 0.0.0.0:32784->443/tcp test-invenio_frontend_1
e23f03eee627 test-invenio-worker "celery worker -A in…" 3 hours ago Up 4 minutes test-invenio_worker_1
ab78abfae152 test-invenio-web-api "uwsgi /opt/test-inv…" 3 hours ago Up 4 minutes 0.0.0.0:32782->5001/tcp test-invenio_web-api_1
79dbebae1d14 test-invenio-web-ui "uwsgi /opt/test-inv…" 3 hours ago Up 4 minutes 0.0.0.0:32783->5000/tcp test-invenio_web-ui_1
943e32add580 mher/flower "flower --broker=amq…" 3 hours ago Up 4 minutes 0.0.0.0:5555->5555/tcp test-invenio_flower_1
af5cc07a67d4 rabbitmq:3-management "docker-entrypoint.s…" 3 hours ago Up 4 minutes 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp test-invenio_mq_1
3ce29066cdb7 postgres:9.6 "docker-entrypoint.s…" 3 hours ago Up 4 minutes 0.0.0.0:5432->5432/tcp test-invenio_db_1
305163ea0a3c redis "docker-entrypoint.s…" 3 hours ago Up 4 minutes 0.0.0.0:6379->6379/tcp test-invenio_cache_1
4e731b8e59bc docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2 "/usr/local/bin/dock…" 3 hours ago Up 4 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp test-invenio_es_1
My instance was cookie-cut with postgres and ES 6.
Include TODO instructions showing how to generate the requirements.txt (i.e. pip freeze)
Should we remove Python 2 support from the cookiecutter template in order to not encourage starting new projects on Python 2.
After setting up instance two things from run-tests.sh are failing that are fixable in the template
foobar/version.py:1 at module level:
D205: 1 blank line required between summary line and description (found 0)
tests/api/conftest.py:14 in public function `create_app`:
D103: Missing docstring in public function
tests/ui/conftest.py:14 in public function `create_app`:
D103: Missing docstring in public function
tests/e2e/conftest.py:14 in public function `create_app`:
D103: Missing docstring in public function
--- /home/krnowak/src/foobar/tests/conftest.py:before 2018-03-23 10:47:26.316118
+++ /home/krnowak/src/foobar/tests/conftest.py:after 2018-03-23 10:57:06.865053
@@ -6,4 +6,3 @@
"""Common pytest fixtures and plugins."""
from __future__ import absolute_import, print_function
-
Other's also fail, but they don't make sense for fresh insance (.e.g manifest check or tests)
Since most running instances are using uWSGI
, these packages should be included in setup.py:
uwsgi
uwsgitop
uwsgi-tools
Depending on inveniosoftware/invenio#3899
Essentially bump Invenio >= 3.1.0dev20181101
Similar to inveniosoftware/cookiecutter-invenio-module we need:
.editorconfig
.gitignore
.travis.yml
AUTHORS.rst
LICENSE
README.rst
cookiecutter.json
docs
hooks
run-tests.sh
The generated documentation for a new site should be checked for consistency.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.