Base package for building Invenio application factories.
Further documentation is available on https://invenio-base.readthedocs.io/
Base package for building the Invenio application.
Home Page: https://invenio-base.readthedocs.io
License: MIT License
Base package for building Invenio application factories.
Further documentation is available on https://invenio-base.readthedocs.io/
The two CLI commands below are not documented:
inveniomanage instance entrypoints
inveniomanage instance migrate-secret-key
Especially secret key migration would need docs - i.e. how should other modules hook into the secret key migration.
Check also API documentation for missing modules.
In Flask v2.2 before_(app_)first_request
was deprecated and will be removed in v2.3.
We are currently using this through various modules for registering menus or other special cases:
before_first_request
before_app_first_request
Blueprint.record_once
(see #171 (comment) ๐ )We should either:
Flask < 2.3
in this moduleinvenio-app
lifecycle to introduce a similar mechanism.Check if actually needed, or we remove it.
param
, returns
, raises
, versionadded
)In order to avoid patterns like this, it would be good if we here can override the url_prefix
via a configuration variable instead.
Co-authored-by: @jirikuncar
The Werkzeug v2.3.0 release, removed many deprecations, some of which are used by or break Invenio modules and their dependencies. More notably so far we've discovered:
Authorization
headers with Bearer tokens, failing to load in Invenio-OAuth2ServerTraceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in full_dispatch_request
rv = self.preprocess_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2313, in preprocess_request
rv = self.ensure_sync(before_func)()
File "/usr/local/lib/python3.9/site-packages/invenio_oauth2server/ext.py", line 189, in verify_oauth_token_and_set_current_user
valid, req = oauth2.verify_request(scopes)
File "/usr/local/lib/python3.9/site-packages/flask_oauthlib/provider/oauth2.py", line 529, in verify_request
return self.server.verify_request(
File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/base.py", line 64, in wrapper
return f(endpoint, uri, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 69, in verify_request
request.token_type = self.find_token_type(request)
File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 85, in find_token_type
estimates = sorted(((t.estimate_type(request), n)
File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/endpoints/resource.py", line 85, in <genexpr>
estimates = sorted(((t.estimate_type(request), n)
File "/usr/local/lib/python3.9/site-packages/oauthlib/oauth2/rfc6749/tokens.py", line 300, in estimate_type
if request.headers.get('Authorization', '').startswith('Bearer'):
TypeError: 'NoneType' object is not callable
For the above reasons we have currently pinned Werkzeug to <2.3.0
.
there are two deprecation warnings concerning ProxyFix class:
tests/test_wsgi.py::test_proxyfix[2-4.3.2.1]
/Users/janek/.virtualenvs/invenio-files/lib/python3.6/site-packages/werkzeug/middleware/proxy_fix.py:88: DeprecationWarning: 'num_proxies' is deprecated as of version 0.15 and will be removed in version 1.0. Use 'x_for' instead.
self.num_proxies = num_proxies
and
/Users/janek/invenio-software/src/invenio-base/invenio_base/wsgi.py:53: DeprecationWarning: 'werkzeug.contrib.fixers.ProxyFix' has moved to 'werkzeug.middleware.proxy_fix.ProxyFix'. This import is deprecated as of version 0.15 and will be removed in 1.0.
return ProxyFix(wsgi_app, num_proxies=app.config['WSGI_PROXIES'])
Only devel build is breaking
Problem:
It should be possible to update the SECRET_KEY if it is compromized.
Design TBD
param
, returns
, raises
, versionadded
)The package contains examples/app.py
example application, but there is no test
for it in tests/test_example_app.py
. It should be added. See existing examples:
Currently we have the following environments where we need a Flask application running:
For all of them we currently create two Flask application factories:
create_app
)/api
)Many core Flask Extensions are initialised twice - once on the UI app and once on the API app. The problem with this approach is in all scenarios we create two Flask applications even if we don't need them. E.g. the Celery application only needs one Flask application. Also the CLI shell
command and many other commands would need just one Application. The CLI run
command will however need the full WSGI application (i.e. the two apps) to run.
It seems to me that our current approach is not ideal, but however also don't have any other ideal solutions.
There doesn't seem to be any way to disable the /static
endpoint generated for each application in invenio_base.app:base_app
. When static_folder is set to None (default value), it generates a path to the instance directory.
This creates an /api/static
endpoint.
Same as:
https://travis-ci.org/inveniosoftware/invenio-base/jobs/656676729
Traceback (most recent call last):
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/travis/build/inveniosoftware/invenio-base/tests/test_wsgi.py", line 109, in appview
return jsonify(data)
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/flask/json.py", line 254, in jsonify
if current_app.config['JSONIFY_PRETTYPRINT_REGULAR'] and not request.is_xhr:
File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
AttributeError: 'Request' object has no attribute 'is_xhr'
Fix also deprecation warning in Travis config:
https://travis-ci.org/inveniosoftware/invenio-base/jobs/656676729/config
Creating a new Invenio instance documentation is wrong. The command actually outputs:
$ inveniomanage instance create test
/Users/lnielsen/src/invenio-base/invenio_base/cli.py:45: UserWarning: This command have been deprecated.
warnings.warn('This command have been deprecated.')
This command have been deprecated. Please use this instead:
pip install cookiecutter
cookiecutter https://github.com/inveniosoftware/cookiecutter-invenio-instance
Note, there's also a spelling mistake above which should also be fixed: This command have been
-> This command has been
.
Probably removing the section is the best, since this should be documented in general Invenio documentation instead.
I tried following the instructions to set up a test site:
conda create -n test-invenio-base python=2.7
source activate test-invenio-base
cd invenio-base/ # (git clone of dev version)
pip install .
cd ../
inveniomanage instance create mysite
cd mysite
ls
pip install .
But the mysite
command don't exist following this (the docs suggest that mysite --help
should work, but I get mysite: command not found
. Am I doing something wrong?
Note that doing python setup.py install
instead of pip install .
doesn't change anything.
Output of pip freeze
if needed:
alembic==0.9.2
amqp==2.1.4
angular-gettext-babel==0.3
arrow==0.10.0
Babel==2.4.0
billiard==3.5.0.2
binaryornot==0.4.3
blinker==1.4
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
cookiecutter==1.5.1
dojson==1.3.1
elasticsearch==2.4.1
elasticsearch-dsl==2.2.0
enum34==1.1.6
Flask==0.12.2
Flask-Alembic==2.0.1
Flask-Assets==0.12
Flask-BabelEx==0.9.3
Flask-Breadcrumbs==0.4.0
Flask-CeleryExt==0.3.0
Flask-Collect==1.2.2
Flask-Limiter==0.9.4
Flask-Menu==0.5.1
Flask-SQLAlchemy==2.2
functools32==3.2.3.post2
future==0.16.0
idna==2.5
invenio-app==1.0.0a1
invenio-assets==1.0.0b6
invenio-base==1.0.0a15.dev20160823
invenio-config==1.0.0b3
invenio-db==1.0.0b7
invenio-i18n==1.0.0b3
invenio-indexer==1.0.0a9
invenio-jsonschemas==1.0.0a4
invenio-marc21==1.0.0a5
invenio-oaiserver==1.0.0a12
invenio-pidstore==1.0.0b1
invenio-query-parser==0.6.0
invenio-records==1.0.0b1
invenio-records-rest==1.0.0a18
invenio-records-ui==1.0.0a9
invenio-rest==1.0.0a10
invenio-search==1.0.0a9
invenio-search-ui==1.0.0a6
invenio-theme==1.0.0b2
itsdangerous==0.24
Jinja2==2.9.6
jinja2-time==0.2.0
jsmin==2.2.2
jsonpatch==1.15
jsonpointer==1.10
jsonref==0.1
jsonresolver==0.2.1
jsonschema==2.6.0
kombu==4.0.2
limits==1.2.1
lxml==3.8.0
Mako==1.0.6
MarkupSafe==1.0
marshmallow==2.13.5
mysite2==0.1.0.dev0
node-semver==0.1.1
ordereddict==1.1
pluggy==0.4.0
poyo==0.4.1
pyPEG2==2.15.2
pyrtlsdr==0.2.3
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
requests==2.17.3
simplejson==3.10.0
six==1.10.0
speaklater==1.3
SQLAlchemy==1.1.10
SQLAlchemy-Utils==0.32.14
urllib3==1.21.1
vine==1.1.3
webargs==1.7.0
webassets==0.12.1
Werkzeug==0.12.2
whichcraft==0.4.1
$ python manage.py --debug shell
Python 3.4.3 (default, Apr 7 2015, 08:05:21)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
App: mysite [debug]
Instance: /Users/lnielsen/envs/invenio3test/bin/../var/mysite-instance
>>> app.debug
True
>>> app.wsgi_app.mounts['/api'].debug
False
>>> app.config['DEBUG']
True
>>> app.wsgi_app.mounts['/api'].config['DEBUG']
False
As cookiecutter
is now based on invenio-app
I think that celery.py
and wsgi.py
should be removed. In any case these two files are broken as they includes factory.py
which has been removed.
I've got a module for registering a "draft" version for each record found in INVENIO_RECORDS_REST
. I'd like to have the draft endpoints to behave exactly the same way as normal invenio record rest endpoints - for example to be able to use url_for('invenio_records_rest.*_item')
and use all the infrastructure created around records.
The only working solution that I've been able to find so far is to, at some point, add those extra records into INVENIO_RECORDS_REST
.
This must be done before records_rest blueprint is created, i.e. before the blueprint loader is invoked in _create_app
.
Currently there is no signal called after extensions are loaded (this is the moment when merged config becomes available) but before blueprint loading.
I was able to find a workaround - in my extension's init_app I replace INVENIO_RECORDS_REST
dictionary with an instance of my own class that creates draft endpoints when getters (such as .items()
) are invoked, but this hack is too fragile as it might be called prematurely by external module trying to access rest configuration.
I'd suggest adding the following extra signals that would work in cooperation with already existing app_created
/app_loaded
called in this order:
app_config_loaded
- called just after loading config via config loader
app_converters_loaded
- called here when converters are loaded
app_extensions_loaded
- called after extensions are loaded and before blueprints are created
The new signals would have the same signature as the existing ones, that is app_*_loaded(_create_app, app=app)
The signals could be used in various cases when one would like to modify the app before the next phase of loading.
For my use case the workaround above currently works. Another alternative would be to copy/paste/adapt invenio startup
sequence to include the modification code and call this instead of invenio's default.
Package version (if known): V12 latest
After changing the SECRET_KEY
in invenio.cfg
and running invenio instance migrate-secret-key --old-key CHANGE_ME
, users are unable to log in with the message "incorrect password". It appears the command accepts any value as old-key
without regenerating the secrets with the new secret key, posing a potential security risk to not be able to access the system after.
SECRET_KEY
in invenio.cfg
.invenio instance migrate-secret-key --old-key CHANGE_ME
.Users should be able to log in with their credentials after the secret key migration, and the process should validate the old-key
properly to ensure secrets are regenerated with the new secret key securely.
The issue was discovered during testing of the migrate-secret-key
command, raising concerns about the command's validation of the old-key
parameter and the regeneration process of secrets.
The following cookiecutter change:
inveniosoftware/cookiecutter-invenio-module#98
should be propagated to this Invenio module.
Namely, in run-tests.sh
, the sphinx for doctests is invoked after pytest run:
$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctest
This sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.
A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build
invocation.
This both solved Travis CI build failures and simplified test suite execution.
Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).
Invenio-base wsgi.py
should be updated for new werkzeug
invenio_base/wsgi.py:36: DeprecationWarning: 'werkzeug.wsgi.DispatcherMiddleware' has moved to 'werkzeug.middleware.dispatcher.DispatcherMiddleware'
See comments in #31
conf_loader
bp_entry_points
ext_entry_points
Do we want to use entry_points to load Flask commands? If yes, shall we use entry_points group flask.commands
or create new configuration option for the entry_point group name in create_cli
factory?
cc @inveniosoftware/triagers
Scenario:
$ pip install invenio-base
$ inveniomanage instance create myapp
Result:
Traceback (most recent call last):
File "/Users/nharraud/.virtualenvs/b2share-scratch/bin/inveniomanage", line 7, in <module>
from invenio_base.manage import main
File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/manage.py", line 25, in <module>
from .factory import create_app
File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/factory.py", line 47, in <module>
from .wrappers import Flask
File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/wrappers.py", line 75, in <module>
from .signals import before_handle_user_exception
File "/Users/nharraud/.virtualenvs/b2share-scratch/lib/python2.7/site-packages/invenio_base/signals.py", line 26, in <module>
from blinker import Namespace
ImportError: No module named blinker
The current locations of instance and folder paths doesn't work nicely when used through python manage.py
.
flask-shell-ipython
is a flask module that changes the default python shell to IPython when running the command flask shell
(or invenio shell
in our case)
It might actually be possible to remove since the only really blocking module we found in terms of relying on application context is invenio-pages
(because it performs an SQL query to fetch the available pages from the database).
Originally posted by @slint in #172 (comment)
Remove manage.py
script from templates and rely only on installed cli
.
Further reading http://click.pocoo.org/5/setuptools/
Improve naming convention of ext_entry_points
and bp_entry_points
(see https://github.com/inveniosoftware/invenio-base/blob/master/invenio_base/app.py#L42-L43)
cc @lnielsen
/
Welcome to {{ site_name }}!
Package version (if known): v1.2.12
Remove werkzeug upper pin from here when flask-login fixes the incompatibility.
Investigate if it is not better to distribute the cookiecutter template separately of Invenio-Base.
Needed for inveniosoftware/invenio-app#66
This task should be built on top of PR #27 (which we can hopefully integrate later today)
Add a command startproject
(name TBC) to inveniomanage
that will generate an initial project structure using cookiecutter
Example of usage:
inveniomanage startproject mysite
will generate something a project structure like this:
https://github.com/lnielsen/mysite
The project structure should be generated using cookiecutter Pythonic API.
The cookiecutter template should be included in invenio-base and full path to found using pkg_resources
.
Change to e.g. depend on e.g. invenio[base,auth,metadata]
. Add docker support? Better documentation?
As discussed:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.