viur-framework / viur-core

The core component of ViUR, a development framework for Google App Engine™.

viur-core

This is the core library component of the ViUR framework.

About

ViUR is an application development toolkit for the Google App Engine™.

ViUR is an open source software development framework that was created to fulfill both designers and developers needs and requirements. It provides a clear concept for implementing agile data management software. It's written in Python™ and already attracted a steady growing community constantly helping and improving ViUR.

Getting started

To get started with ViUR, check out viur-base. It comes with a pre-configured and well documented project template to immediately start with.

Migration

from <=v3.5 to v3.6

In #833 the config has changed from a dict to an object. To migrate the access expressions like conf["option"] in your project to conf.option the viur-core provides a migration script. Install the viur-core in your project, open a (virtual) environment shell and viur-core-migrate-config will be available. After checking the result with viur-core-migrate-config ./deploy/ -d you can apply the changes with viur-core-migrate-config ./deploy/ -x.

Contributing

Help of any kind to extend and improve or enhance this project in any kind or way is always appreciated.

We take great interest in your opinion about ViUR. We appreciate your feedback and are looking forward to hear about your ideas. Share your vision or questions with us and participate in ongoing discussions.

See our contribution guidelines for details.

License

Copyright © 2024 by Mausbrand Informationssysteme GmbH.
Mausbrand and ViUR are registered trademarks of Mausbrand Informationssysteme GmbH.

You may use, modify and distribute this software under the terms and conditions of the GNU Lesser General Public License (LGPL). See the file LICENSE provided within this package for more information.

viur-core's Issues

Rename clearUpdateTag to updateRelations

Hello!

The name clearUpdateTag=True/False parameter that can be given to Skeleton.toDB() is misleading. Renaming this to updateRelations would be more understandable, so that updateRelations=True is the default, updateRelations=False can be set if not relation update should be invoked.

userBone(creationMagic=True) throws annoying error when no user is logged-in

A bone configured creator = userBone(creationMagic=True) throws this error when userBone.py in case the user is not logged in wants to set the bone to None. The error message is annoying and doesn't make much sense. In general, it should be considered that setting a bone to None isn't required in ViUR3 anymore in case the skeleton was freshly instanciated.

You must supply exactly one Database-Key to creator
Traceback (most recent call last):
  File "/workspace/viur/core/request.py", line 219, in processRequest
    self.findAndCall(path)
  File "/workspace/viur/core/request.py", line 392, in findAndCall
    res = caller(*self.args, **self.kwargs)
  File "/workspace/modules/user.py", line 294, in trigger
    "sendeZeitPunkt": sendezeitpunkt
  File "/workspace/modules/chatmessage.py", line 352, in _importMessageFromAdnova
    assert chatroomSkel.toDB()
  File "/workspace/skeletons/feedlist.py", line 92, in toDB
    return super().toDB(skelValues, clearUpdateTag)
  File "/workspace/abstracts/subscriptable.py", line 76, in toDB
    return super().toDB(skelValues, clearUpdateTag)
  File "/workspace/viur/core/skeleton.py", line 887, in toDB
    _bone.performMagic(skelValues, bkey, isAdd=isAdd)
  File "/workspace/viur/core/bones/userBone.py", line 28, in performMagic
    return self.setBoneValue(skel, key, None, False)
  File "/workspace/viur/core/bones/relationalBone.py", line 856, in setBoneValue
    raise ValueError("You must supply exactly one Database-Key to %s" % boneName)
ValueError: You must supply exactly one Database-Key to creator

Invite user process

When you create a new user in ViUR, it would be a good idea to send this user an email with a one-time password and oblige him/her to change this password at the first login.

relationalBone(multiple=True) but with unique relations per destination

Hi there,

I'm having a requirement for some kind of unique-Feature for relationalBone(multiple=True) bones, where multiple relations to different destination can be added, but only one relation for each target key is allowed.
In the current implementation, I can create multiple relations to the same destination entity.

How could this be implemented?
How could it be named? (I think unique is not the right wording, because it is used differently).

Implement sortIndexBone

As decided in the ViUR-round at 2020-08-07 we want the sortIndexBone.
(should be implemented with append/amend)

Deleting multiple entries at once from the Vi throws status 412 errors

possibly because the core cannot serve new skeys in rapid succession. could also be a problem with the Vi itself... not sure if this is the correct place or if the issue concerns both core and vi

Getting rid of collectSkelData

Hello,

when calling a module's view-function, the variable skel is directly pushed as a Skeleton-object into the template (see here and below).

But when inside of a template the getSkel()-function is used, no Skeleton object is returned, but a dict containing the values that is generated by render.html.default.Render.collectSkelData() (see here).

This is not a consequent behavior, and when Skeleton objects are considered to be replaced by dicts, it should be the case everywhere. There are several calls to collectSkelData in the entire HTML renderer, which might be needles, including the whole function.

Implement/Port bone specific rendering

viur-framework/server#122

Implement/Port cache tracking feature from server

https://github.com/viur-framework/server/tree/feature/cacheTracking

Generalize bone.fromClient() for multiple- and language-bones

Hello,

I'm currently implementing a general ViUR export & importer for Google Spreadsheets. I found out, that it is a good choice to build pathes to values, like "relbone.key.0", "relbone.key.1" or also "txtbone.de" and "txtbone.en". This is very straightforward and works both to access values from a JSON-rendered structure but also to specify values for the bone.fromClient() function.

But then, I found out that when I name a multiple-string bone this way "strbone.0", "strbone.1", etc. it is not recognized by ViUR because it looks for a list of value in "strbone" only.

My proposal here would be to allow for both: a list may be available under "strbone", or the single entries could be specified as "strbone.0", "strbone1." etc. This logic is also the same for language-bones which are multiple, here the case is "strbone.en.0", "strbone.en.1" containing single values or "strbone.en" allowing for a list. Note, that for relationalBone(multiple=True), this is done exactly this way, so there is a asymmetry between how stringBone() and how relationalBone() values are parsed.

This issue is only a proposal for a pull request that will follow when I have some time. I generally would like to revise all bone.fromClient() functions to work equally with the same notation. In my opinion, there is a lot of code in these functions which is very redudnant and could be cleaned up this way (regard, f.e. stringBone, which has branches for if bone.multiple and not bone.languages or if not bone.multiple and bone.languages... this is only both awkward and error-prone.

File Upload broken?

File upload seems to be broken. Removing the timeout parameter allows the upload. Is this parameter still needed here?
/deploy/viur/core/modules/file.py", line 296, in initializeUpload
uploadUrl = blob.create_resumable_upload_session(content_type=mimeType, size=size, timeout=60)
TypeError: create_resumable_upload_session() got an unexpected keyword argument 'timeout'

Rename rawValueBone to just rawBone

We have a stringBone, a relationalBone, a fileBone... and now a rawValueBone? Why isn't this just be called rawBone, to comply with the already existing naming convention? Or, let's rename stringBone, relationalBone, fileBone ... into stringValueBone, relationalValueBone and fileValueBone...

The type should be changed to "raw" instread of "rawvalue" as well.

Bug: Datebones substract a day on save

When I save an entry with a dateBone, the dateBones date is lowered by one day on each save.

This has to be a similar problem like we had with the drifting times (which also drifted an hour into the past on save)...

Deferred Tasks not serializing the current user

Probably due to getCurrentUser returning the skeleton now

changing the default renderer ist not possible

The core setup function allows to switch the html renderer to another renderer. The feature is currently broken.

main.py:
app = core.setup(modules, render,"json")

File "/tmp/tmp6NvlJv/lib/python3.8/site-packages/viur/core/__init__.py", line 252, in setup conf["viur.mainApp"] = buildApp(modules, render, default) File "/tmp/tmp6NvlJv/lib/python3.8/site-packages/viur/core/__init__.py", line 185, in buildApp setattr(getattr(res, renderName), moduleName, obj) AttributeError: 'bool' object has no attribute '_tasks'

Lock .toDB, .delete etc. on skeletons if an output render is attached

Maybe related to #187.

differentiate between Unauthorized and Forbidden

In several placed in core (especially in the edit-methods) we are currently raising an Unauthorized exception.
But unauthorized means you have submitted no or invalid credentials - If you are logged in and have not the move-/edit-access it's simply just forbidden to you.
So we have to differentiate between not logged in (Unauthorized) and no access (Forbidden).

Originally annotation: https://github.com/viur-framework/viur-core/pull/28/files/dd4287cc596af96f9d39d02c24621339134f098c..1b34887b40e9d3ee6baa71fb4f69771a1a9339f5#r437333030

Reenable changelist evaluation in updateRelations

Disabled as a temporary work-a-round in #286

Commit 0b84619b82fe1ccb366c89fa55d69657516605b3 breaks our local application.

Hey,

The commit 0b84619b82fe1ccb366c89fa55d69657516605b3 breaks my system (viur-core-logs below) when running it locally on my Arch Linux machine.

The problem is, that the module cloud, (import in /deploy/viur/core/modules/) does not exist. May I miss something while initializing of the project or the development environment?

[2021-02-12 13:16:29 +0100] [97405] [INFO] Starting gunicorn 20.0.0
[2021-02-12 13:16:29 +0100] [97405] [INFO] Listening at: http://0.0.0.0:23908 (97405)
[2021-02-12 13:16:29 +0100] [97405] [INFO] Using worker: threads
[2021-02-12 13:16:29 +0100] [97408] [INFO] Booting worker with pid: 97408
WARNING  2021-02-12 13:16:30,273 tasks.py:58] Taskqueue disabled, tasks will run inline!
DEBUG    2021-02-12 13:16:30,273 _default.py:180] Checking store_credentials.json for explicit credentials as part of auth process...
[2021-02-12 13:16:30 +0100] [97408] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
    super().init_process()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/workers/base.py", line 133, in init_process
    self.load_wsgi()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/tmp/tmpccI_eR/lib/python3.9/site-packages/gunicorn/util.py", line 331, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<censored>/deploy/main.py", line 28, in <module>
    from viur.core.modules.file import thumbnailer
  File "<censored>/deploy/viur/core/modules/file.py", line 30, in <module>
    from google.cloud import iam_credentials_v1
ImportError: cannot import name 'iam_credentials_v1' from 'google.cloud' (unknown location)
[2021-02-12 13:16:30 +0100] [97408] [INFO] Worker exiting (pid: 97408)
Waiting up to 5 seconds.
DEBUG    2021-02-12 13:16:30,429 requests.py:181] Making request: POST https://oauth2.googleapis.com/token
DEBUG    2021-02-12 13:16:30,430 connectionpool.py:971] Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG    2021-02-12 13:16:30,543 connectionpool.py:452] https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG    2021-02-12 13:16:30,625 background_thread.py:123] Submitted 1 logs
DEBUG    2021-02-12 13:16:30,721 background_thread.py:123] Submitted 2 logs
DEBUG    2021-02-12 13:16:30,721 background_thread.py:160] Background thread exited gracefully.
Sent all pending logs.
[2021-02-12 13:16:30 +0100] [97405] [INFO] Shutting down: Master
[2021-02-12 13:16:30 +0100] [97405] [INFO] Reason: Worker failed to boot.

/{module}/view/structure is broken in json/vi renderer

The requests ends in a the 401 Unauthorized.

The problem is the canView-method. The passing skeleton is empty/clean. Therefor skel["key"] is None and is declared as invalid in buildDBFilter of the KeyBone (called from mergeExternalFilter).

ERROR    2021-02-26 18:30:59,117 keyBone.py:100] None could not be converted to bytes
Traceback (most recent call last):
  File "/.../deploy/viur/core/bones/keyBone.py", line 98, in _decodeKey
    return KeyClass.from_legacy_urlsafe(key)
  File "/tmp/tmpfc49Z1/lib/python3.9/site-packages/google/cloud/datastore/key.py", line 356, in from_legacy_urlsafe
    urlsafe = _to_bytes(urlsafe, encoding="ascii")
  File "/tmp/tmpfc49Z1/lib/python3.9/site-packages/google/cloud/_helpers.py", line 370, in _to_bytes
    raise TypeError("%r could not be converted to bytes" % (value,))
TypeError: None could not be converted to bytes
WARNING  2021-02-26 18:30:59,120 keyBone.py:101] Could not decode key None
ERROR    2021-02-26 18:30:59,120 db.py:326] 
Traceback (most recent call last):
  File "/.../deploy/viur/core/bones/keyBone.py", line 98, in _decodeKey
    return KeyClass.from_legacy_urlsafe(key)
  File "/tmp/tmpfc49Z1/lib/python3.9/site-packages/google/cloud/datastore/key.py", line 356, in from_legacy_urlsafe
    urlsafe = _to_bytes(urlsafe, encoding="ascii")
  File "/tmp/tmpfc49Z1/lib/python3.9/site-packages/google/cloud/_helpers.py", line 370, in _to_bytes
    raise TypeError("%r could not be converted to bytes" % (value,))
TypeError: None could not be converted to bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../deploy/viur/core/bones/keyBone.py", line 124, in buildDBFilter
    dbFilter.filter("%s%s =" % (prefix or "", KEY_SPECIAL_PROPERTY), _decodeKey(rawFilter[name]))
  File "/.../deploy/viur/core/bones/keyBone.py", line 102, in _decodeKey
    raise RuntimeError()
RuntimeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../deploy/viur/core/db.py", line 321, in mergeExternalFilter
    bone.buildDBFilter(key, skel, self, filters)
  File "/.../deploy/viur/core/bones/keyBone.py", line 128, in buildDBFilter
    raise RuntimeError()
RuntimeError

Email Bone must not be empty, even if required=False is set.

Email Bone must not be empty, even if required=False is set.

baseBone-fallback for a recordBone failing with "'NoneType' object is not iterable" when None

In this case, the recordBone usage additional_addresses = recordBone() produces this error when a fresh, empty skeleton shall be saved.

Traceback (most recent call last):
  File "/workspace/viur/core/request.py", line 221, in processRequest
    self.findAndCall(path)
  File "/workspace/viur/core/request.py", line 395, in findAndCall
    res = caller(*self.args, **self.kwargs)
  File "/workspace/modules/user.py", line 162, in login
    assert skel.toDB()
  File "/workspace/skeletons/user.py", line 349, in toDB
    return super().toDB(skel, clearUpdateTag)
  File "/workspace/viur/core/skeleton.py", line 883, in toDB
    key, dbObj, skel, changeList = db.RunInTransaction(txnUpdate, key, skelValues, clearUpdateTag)
  File "/workspace/viur/core/db.py", line 1007, in RunInTransaction
    res = callee(*args, **kwargs)
  File "/workspace/viur/core/skeleton.py", line 695, in txnUpdate
    bone.serialize(skel, key, True)
  File "/workspace/viur/core/bones/bone.py", line 408, in serialize
    for singleValue in newVal:
TypeError: 'NoneType' object is not iterable

Hide "older" entries from not being updated by updateRelations

Similar to the updateLevel flag at relationalBones, it might be useful to provide generally a generic way to mark entities as "too old to be updated" at some point, so these entries won't be updated by the updateRelations task.

Following use-case: There is an database entry representing a task. The task's relationalBones should be updated all the time when something changes. At some point, the task is marked as "completed". And 1 year later, nobody will every take a view into this task again - so its relationalBones are not worth to be kept updated. Therefore, the entry is marked as "archived" or some kind, and updateRelations won't match it anymore.

skel.items() in Jinja always delivers empty values

Hello!

This view-Template does only work when bones are accessed by skel[boneName]. The values item of the iterator is always empty.

{% extends "viur_base.html" %}

{% block content %}
	{% for key, value in skel.items() %}
		value always Emtpy: {{ key }} = {{ value }}<br>
		this perfectly Works: {{ key }} = {{ skel[key] }}<br>
	{% endfor %}
{% endblock %}

Migrating from Google Sign-In to Identity Services

https://developers.google.com/identity/gsi/web/guides/migration?hl=en

Is the rootNode convention still used?

We have fixed a bug #196 and found that rootNode is still used in some places e.g. fileNodeSkel and there is a fixme on line 377

Is this going to be fixed soon?

Standardize several notations

There are several places in ViUR where the same functionality or behavior has different notations, but conceptually means the same. This issue should be first used as a collector for these notations, which finally should be renamed or equalized to the same notation when version 3.0 is finalized, to avoid confusions especially for newcomers.

limit, amount

Problem

mergeExternalFilter() uses amount as GET-parameter, but sets the limit of the query. Functions like run() and fetch() are using limit instead.

Proposal

Use limit everywhere.

getSkel(), getEntry()

Problem

In the HTML-renderer, getSkel() returns the structure, and getEntry() returns and entry (which normally is a skel!). There is also a getSkel() function in db.Query which returns a Skeleton, and not the structure.

Proposal

• Rename getSkel() to getStructure()
• Rename getEntry() to getSkel()

Edit: This has been fixed with #48
~~

add, edit, delete => onItemAdded, onItemEdited, onItemDeleted => addItemSuccess(), editItemSuccess(), deleteSuccess()

Problem

Why can't we call them just "edit", "add" and "delete" same as the actions they do?

Proposal

My proposal is to rename this

~~edit => self.onItemEdited(skel) => self.render.editItemSuccess(skel)~~
~~add => self.onItemAdded(skel) => self.render.addItemSuccess(skel)~~
~~delete => self.onItemDeleted(skel) => self.render.deleteSuccess(skel)~~

into this:

~~edit => self.onEdit(skel) => self.render.editSuccess(skel)~~
~~add => self.onAdd(skel) => self.render.addSuccess(skel)~~

~~ delete => self.onDelete(skel) => self.render.deleteSuccess(skel)~~

Edit: This was fixed by #39

Variants of relationalBone have misleading names

Problem

Why are specialized versions of relationalBone not correctly sub-classed... treeDirBone(), treeItemBone(), fileBone()
See here...

Proposal

• treeDirBone() indicates itself as relational.treedir, should be relational.tree.node
• treeItemBone() indicates itself as relational.treeitem, should be relational.tree.leaf and renamed into treeLeafBone()
• ... it makes then sense to provide a treeItemBone() which is just relational.tree to select both nodes and leafs...
• fileBone() indicates itself as treeitem.file??????, should be relational.tree.leaf.file
• hierarchyBone() indicates itself as just relational without speciality - IMHO this can be removed entirely and replaced by relationalBone().

Edit: This has been fixed by #49

Render type specifier needed

Every renderer should provide a type specifier, so that one can do

if self.render.kind = "json" or self.render.kind.startswith("json."):
    # let module behave different for this renderer

These kinds specifiers should be

• html => "html"
• json => "json"
• xml => "xml"
• vi => "json.vi"
• admin => "json.admin"

Otherwise, I always have to import the specific renderer and make an isinstance() check on it.

missing dependency

I am not sure if something has changed on the SDK side, but without this entry in my requirements I now get an error when starting a project with dev_appserver (or when deployin to GAE)

For now this can just be added to the project requirements as a quick fix

googleapis-common-protos[grpc]==1.53.0 --hash=sha256:c075eddaa2628ab519e01b7d75b76e66c40eaa50fc52758d8225f84708950ef2

user-module must be exposed for html renderer to use getCurrentUser

getCurrentUser uses the user-module in the default renderer context. But if there is no user-module in the html-renderer (which is mostly the default render) this method will always return None, which is quite unusable. Therefore the user-module must always be exposed for the html-renderer, even if you want not use the user-html-templates.

viur-core/utils.py

Lines 199 to 213 in fc73b29

	def getCurrentUser():
	"""
	Retrieve current user, if logged in.
	If a user is logged in, this function returns a dict containing user data.
	If no user is logged in, the function returns None.
	:rtype: dict | bool
	:returns: A dict containing information about the logged-in user, None if no user is logged in.
	"""
	user = None
	if "user" in dir(conf["viur.mainApp"]): # Check for our custom user-api
	user = conf["viur.mainApp"].user.getCurrentUser()
	return user

Obey/choose a better color profile when converting from CMYK to RGB

defaultValue is not handled as expected

Hello there,

this issue relates to two different problems relating defaultValue on bones.

1. I want to set a relationalBone's defaultValue using a callable, which returns either None or a valid key. Pull request #94 resolves the issue for my case, but I'm sure that its not satisfying ViUR's concept. FIXED AND REPLACED BY #97
2. When I instanciate a skeleton, the defaultValues are not pre-filled as expected. I tried to handle this case by adding a function

	def initialize(self):
		"""
		Initialize this instance with all its default values.
		"""
		for key, bone in self.boneMap.items():
			defaultValue = bone.getDefaultValue()
			if defaultValue is not None:
				self.accessedValues[key] = defaultValue

		return self

directly to skeleton.SkeletonInstance.

A call would look like so:

	def addSkel(self):
		skel = super().addSkel().clone().initialize()
		...

It curiously works, and the entity is also saved, but I'm getting an error 500 anyway with this content:

ERROR    2020-08-04 23:48:46,137 request.py:245] Viur caught an unhandled exception!
ERROR    2020-08-04 23:48:46,139 request.py:246] 400 At most 20 nested array/entity values are supported.
Traceback (most recent call last):
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "At most 20 nested array/entity values are supported."
        debug_error_string = "{"created":"@1596577726.137116011","description":"Error received from peer ipv4:172.217.22.106:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"At most 20 nested array/entity values are supported.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/MyProject/deploy/viur/core/request.py", line 218, in processRequest
    self.findAndCall(path)
  File "~/MyProject/deploy/viur/core/request.py", line 392, in findAndCall
    res = caller(*self.args, **self.kwargs)
  File "~/MyProject/deploy/viur/core/prototypes/list.py", line 256, in add
    self.onAdded(skel)
  File "~/MyProject/deploy/modules/chatmessage.py", line 59, in onAdded
    conf["viur.mainApp"].chatthread.updateCache(
  File "~/MyProject/deploy/modules/chatthread.py", line 73, in updateCache
    db.RunInTransaction(txn)
  File "~/MyProject/deploy/viur/core/db.py", line 920, in RunInTransaction
    res = callee(*args, **kwargs)
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/cloud/datastore/batch.py", line 302, in __exit__
    self.commit()
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/cloud/datastore/transaction.py", line 241, in commit
    super(Transaction, self).commit()
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/cloud/datastore/batch.py", line 274, in commit
    self._commit()
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/cloud/datastore/batch.py", line 249, in _commit
    commit_response_pb = self._client._datastore_api.commit(
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 570, in commit
    return self._inner_api_calls["commit"](
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    return retry_target(
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/tmp/tmpjDioH2/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 At most 20 nested array/entity values are supported.

I need a valid, ViUR-compatible solution for boths cases, because they partly stick together.

Limit multiple-bones to maximum amount of entries

It might be useful to configure bones as multiple, but limit the amount of entries allowed. Especially when working with subSkels, this would be essential: maybe one kind of the subSkel allowes for only one entry, where others allow for more. Configuring the bone in the addSkel or editSkel differently might run into problems, as there might be entites with either lists or single entries.

An implementation might look like this:

unlimited = stringBone(multiple=True)
limitedTo3 = stringBone(multiple=3)

This is just an idea that should be discussed at some time.

Implement relation depth parameter

As decided in the ViUR-round at 2020-08-07 we need a parameter to set the depth of refKeys in relations.

Sometimes getting 502 Bad Gateway on GAE deployed apps

I'm ocassionally getting a 502 Bad Request with my project.

Deep in the logs, there's this traceback coming up:

Traceback (most recent call last): File "/layers/google.python.pip/pip/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/layers/google.python.pip/pip/gunicorn/workers/gthread.py", line 92, in init_process super().init_process() File "/layers/google.python.pip/pip/gunicorn/workers/base.py", line 133, in init_process self.load_wsgi() File "/layers/google.python.pip/pip/gunicorn/workers/base.py", line 142, in load_wsgi self.wsgi = self.app.wsgi() File "/layers/google.python.pip/pip/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/layers/google.python.pip/pip/gunicorn/app/wsgiapp.py", line 49, in load return self.load_wsgiapp() File "/layers/google.python.pip/pip/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp return util.import_app(self.app_uri) File "/layers/google.python.pip/pip/gunicorn/util.py", line 331, in import_app mod = importlib.import_module(module) File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/srv/main.py", line 159, in <module> import modules, render File "/srv/modules/__init__.py", line 9, in <module> import viur.core.prototypes.basic File "/srv/viur/core/prototypes/__init__.py", line 2, in <module> from .basic import BasicApplication File "/srv/viur/core/prototypes/basic.py", line 3, in <module> from viur.core.skeleton import skeletonByKind File "/srv/viur/core/skeleton.py", line 509, in <module> class Skeleton(BaseSkeleton, metaclass=MetaSkel): File "/srv/viur/core/skeleton.py", line 377, in __init__ "Skeletons must be defined in a folder listed in conf[\"viur.skeleton.searchPath\"]") NotImplementedError: Skeletons must be defined in a folder listed in conf["viur.skeleton.searchPath"]

We recognized this problem also with another project, so it's not directly project specific. I'm afraid that this problem might come up later when multiple instances are spawning in a high-scalable appliance and it occasionally comes to this error.

Filter by key of subordinated skeleton.

Hi :)

Can someone help me with a problem, please.

I am using a skeleton with a relational bone and I want to use the ListFilter method of the associated module to filter by keys of the subordinated skeleton.

So I tried this line of code to archive this:

query.filter("venue.dest.key", "ahBydWhycG90dG1ldGFsbGVychILEgV2ZW51ZRiAgIDY2v2VCgw")

Also I tried other combinations with "__key__" instead of "key" and/or without ".dest". The result is always empty.
Filtering by other bone values than the key works properly.

What am I doing wrong?

My version of viur-core is 3af1fd303ae37810b3917746a0299940469dd093.

Fixing default downloadUrl

Hello!

Today we found about this bug, and I wanted to issue it here. Due to the caseSensitive=True-configuration of the name bone of fileBaseSkel, the DB-entity-object is used to construct the the downloadURL, rather then the correct value.

Here's a diff which temporarily fixes the problem.

diff --git a/modules/file.py b/modules/file.py
index 5f371db..6793e42 100644
--- a/modules/file.py
+++ b/modules/file.py
@@ -33,8 +33,7 @@ bucket = client.lookup_bucket("%s.appspot.com" % projectID)
 class injectStoreURLBone(baseBone):
        def unserialize(self, skel, name):
                if "dlkey" in skel.dbEntity and "name" in skel.dbEntity:
-                       skel.accessedValues[name] = utils.downloadUrlFor(skel.dbEntity["dlkey"], skel.dbEntity["name"],
-                                                                               derived=False)
+                       skel.accessedValues[name] = utils.downloadUrlFor(skel.dbEntity["dlkey"], skel.dbEntity["name"]["val"], derived=False)
                        return True
                return False

@tsteinruecken told me that it would be better to be solved this differently in viur-core.

Render error messages on /json/... as JSON?

As HTML/Jinja is the default output render, it is okay that it shows error messages in an HTML-style error message:

But why does a /json/... or /vi/...-call also return the same, HTML-style error message and not some piece of JSON, especially regarding error descriptions which are being lost there.

{
    "code": 404,
    "message": "Not Found",
    "description": "The requested resource could not be found."
}

Such a machine-reasable JSON data would be very useful to report back reasons why an action failed back to the user.

Additionally, this ViUR2-patch would also be reasonable by sending the error message via the response header, and allow Ajax-requests to also deal with error messages in case the returned data should explicitly be HTML.

--- a/__init__.py
+++ b/__init__.py
@@ -449,6 +449,11 @@ class BrowseHandler(webapp.RequestHandler):
                                raise
                        self.response.clear()
                        self.response.set_status(e.status, e.descr)
+
+                       # Set machine-readable x-viur-error response header in case there is an exception description.
+                       if e.descr:
+                               self.response.headers["x-viur-error"] = e.descr.encode("utf-8") if isinstance(e.descr, unicode) else str(e.descr)
+
                        res = None
                        if conf["viur.errorHandler"]:
                                try:

Make index to a BasicApplication or not (Reminder to evaluate)

Currently the index-module mostly inherits from object (and the renderer is assigned as attribute) or from the defaultRenderer.
IMO it's much more proper to inherit the index-module from the BasicApplication and handle it like any other module.
This caused also to handle the index differently than a module. See here: viur-framework/server#191

We have to test was is currently possible on a index-module, and what isn't (deferred-tasks, periodicTasks, ...).
Maybe is was only a problem in ViUR 2...

Port XML-renderer to ViUR 3

Since ViUR 2.x the xml-renderer is broken and needs to be ported to the actually API.

pytz.timezone in the arguments of @callDeferred Fuctions

If you pass "pytz.timezone" or "currentRequestData" as argument in a function and then call it with callDeferred you get a TypeError.
This error occurs because the function "preprocessJsonObject" cannot translate the argument correctly.

Possible fix:
elif isinstance(o,pytz.BaseTzInfo): return str(o)

In

viur-core/core/tasks.py

Line 23 in 579fc78

def preprocessJsonObject(o):

htmlSerializer twice

Hey there,

we have currently in modules the htmlSerializer.
First of all it's only a class, not a really ViUR-Module. Furthermore we have another htmlSerializer class in the texBone. I thing one good implementation should be enough.

Using of Infilter raises an error (possible bug)

Hi guys :)

I tried to hand over a tuple or list in combination with the IN filter which raises an error.
The query used in the listFilter method of my module is query.filter("days in", [2 ,3] ).

By setting the imit parameter of the qry.fetch() function call in the _runSingleFilterQuery method of the db.py (line 696) manually to e.g. 5, the IN filter works properly.

May you please provide a solution for this?

Traceback (most recent call last):
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/request.py", line 221, in processRequest
    self.findAndCall(path)
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/request.py", line 395, in findAndCall
    res = caller(*self.args, **self.kwargs)
  File "/home/klaus/coding/ruhrpottmetaller/deploy/modules/event.py", line 78, in index
    return self.list(*args, **kwargs)
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/prototypes/list.py", line 169, in list
    res = query.fetch()
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/db.py", line 877, in fetch
    dbRes = self.run(limit)
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/db.py", line 826, in run
    res.append(self._runSingleFilterQuery(singleQuery, limit))
  File "/home/klaus/coding/ruhrpottmetaller/deploy/viur/core/db.py", line 697, in _runSingleFilterQuery
    res = next(qryRes.pages)
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 243, in _page_iter
    page = self._next_page()
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/cloud/datastore/query.py", line 536, in _next_page
    response_pb = self.client._datastore_api.run_query(
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 383, in run_query
    return self._inner_api_calls["run_query"](
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    return retry_target(
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/tmp/tmpLtdUtG/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Limit must be non-negative.

raise NotFound instead of NotAcceptable

The sever and core raise here (and maybe in other methods as well, not checked yet) a wrong exc. when the entry wasn't found.

viur-core/prototypes/list.py

Lines 203 to 204 in 06a47b1

	if not skel.fromDB(key):
	raise errors.NotAcceptable()

(Creating this issue was faster than a PR. Tbd...)

Cloning a SkeletonInstance with data stored into a relationalBone fails

This works:

# This works
skel = FeedSkel().clone()
skel = skel.clone()

# This works also
skel = FeedSkel().clone()
skel.setBoneValue("name", "Hello World")
skel = skel.clone()

But this does not work:

skel = FeedSkel().clone()
skel.setBoneValue("feedlist", "aglteVByb2plY3RyFQsSCGZlZWRsaXN0GICAgNiGpYMKDA")
skel = skel.clone()

and fails with the following backtrace:

ERROR    2020-07-21 16:36:01,245 request.py:245] Viur caught an unhandled exception!
ERROR    2020-07-21 16:36:01,245 request.py:246] '__deepcopy__'
Traceback (most recent call last):
  File "~/myProject/deploy/viur/core/request.py", line 218, in processRequest
    self.findAndCall(path)
  File "~/myProject/deploy/viur/core/request.py", line 392, in findAndCall
    res = caller(*self.args, **self.kwargs)
  File "~/myProject/deploy/modules/index.py", line 21, in test
    skel = skel.clone()
  File "~/myProject/deploy/viur/core/skeleton.py", line 184, in clone
    res.accessedValues = copy.deepcopy(self.accessedValues)
  File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 151, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
  File "~/myProject/deploy/viur/core/skeleton.py", line 161, in __getattr__
    return self.boneMap[item]
KeyError: '__deepcopy__'

Accessing RelSkel values the old way is not possible

Hello,
in the past I was able to access a relationalBones' destination values this way:

tokSkel["user"]["dest"]["key"]

Due to the new SkeletonValues class, this is not possible anymore. I did change it to according to the following diff, but this is only a work-around, and I know that it's bad.

--- a/skeleton.py
+++ b/skeleton.py
@@ -70,6 +70,12 @@ class SkeletonValues(object):
                self.accessedValues = {}
                self.renderAccessedValues = {}
 
+       def __getitem__(self, item):
+               if item in self.accessedValues:
+                       return self.accessedValues[item]
+
+               return self.entity.get(item)
+
 
 class BaseSkeleton(object, metaclass=MetaBaseSkel):
        """

We definitely need a better solution for this! ViUR 2 projects must be held maintainable even when updated to ViUR 3.

Provide a utils.unescapeString function

3dfd5bd introduced a utils.unescapeString function in the past, and 438fd45 reverted it. Why?

3dfd5bd was made because there was often the case that a stringBone or textBone value contained escaped special characters. This is okay, but there are many cases where the true value is needed, e.g. when it is passed to an interface or rendered by a flare-application which doesn't care about any special characters as they are inserted as TextNodes. In this case, the unescaping function has to be implemented for every case again, again and again. Why was this already existing function be removed?

Either utils.unescapeString will be made available again or the escaping of characters in stringBone and textBone should be removed.

Form Javascript is missing

The javascript merging the date and time select inputs form https://github.com/viur-framework/viur-core/blob/master/template/editform_bone_date.html into the format the server expects on post is broken (https://github.com/viur-framework/viur-core/blob/master/resources/js/viur-forms.js). Relates to #187 .

Rename db.Query.mergeExternalFilter into db.Query.updateFilter

db.Query.mergeExternalFilter is almost used in cases where it hasn't to do anything with an "external" filtering. The naming is misleading. Please rename this function before ViUR3 final. db.Query.updateFilter is an appropriate naming because it requests a dict and updates the provided Query to it.

Unify key handling

In many cases, we do something like this

str(skel["key"])

in many, many situations inside ViUR2 project code to ensure that a key is passed or made available in a serialized form. This does not work anymore with ViUR3! Instead a string like <Key('appconf', 'appconf-modulekey'), project=myproject> is returned.

The function Key.to_legacy_urlsafe() (https://googleapis.dev/python/datastore/latest/keys.html#google.cloud.datastore.key.Key.to_legacy_urlsafe) does not work properly because the prefix is not provided. Also, a decision must be taken in every case before wether the key is a string, a db.Key() or a key's id or name value.

In my opinion, a ViUR-owned, internal Key wrapper must be implemented to handle this case properly and securely in all situations. This would also make forks of ViUR working on other databases easier to implement.