marrow / mongo Goto Github PK
View Code? Open in Web Editor NEWLight-weight utilities and declarative schema (mutable mapping) to augment, not replace the Python MongoDB driver.
Home Page: https://mongo.webcore.io
License: MIT License
Light-weight utilities and declarative schema (mutable mapping) to augment, not replace the Python MongoDB driver.
Home Page: https://mongo.webcore.io
License: MIT License
Document maintains state
described by a cyclic directed graph of allowable string tag-based state transitions.
The current extension in the "shop" fails badly; possibly this is built-in now? Should jump to the diff for that revision.
Given they have an _id
key.
Add the following classmethod
s to the Document
class:
create_collection
get_collection
create_indexes
On this page: https://mongo.webcore.io/installation.html
You have a link towards the bottom for the contributing guide. If clicked, this attempts to take you to https://mongo.webcore.io/CONTRIBUTING.md which appears to hang indefinitely by the spinning icon at the top of the page. In the javascript console, you get
GET https://mongo.webcore.io/CONTRIBUTING.md 404 ()
We can begin the search from the line pulling in the record (.find_one
, &c.) through all reachable points from there, looking for all attribute and dictionary-like access to that object.
¹ At runtime.
Hey, hi!
I really liked the Marrow Mongo approach.
We have been bitten several times by mongoengine and were considering migrating to something else...
I just want to learn more about the project and see if is a fit for us. I am sorry for the trouble!
Thanks and congratulations!
I appear to have a case where Pypy (5.4.1) does not interpret a MutableMapping.register()
'd class as a mapping as per CPython 2/3 on the same codebase as seen by PyMongo. (A… complex case.)
Usage of manual ABC registration is the primary issue; avoiding the indirection of the wrapped objects avoids the symptom in follow-up testing. Subclassing the ABC does work correctly in this particular instance.
This does not correct the overall issue with manual registration, however, as I have classes with incompatible metaclasses that I also wish to participate in the ABC. (With prior Travis failures and work-arounds for the previously encountered incidents of this.). However, Ops is a biggie.
local
alias (identify server-local timezone), 'naive' alias (strip timezone), timezone name, explicit tzinfo
instance, or use of callback returning one of these.Existing work in c0cb7d2
A String field specialization utilizing pathlib.PurePosixPath
conversion.
Getting this idea out of my brain onto "paper" while it's fresh, permit the return of scalar attribute proxies from find_one()
, that is, an apparent instance of the targeted Document
sub–class that upon access to an attribute (field descriptor) or subscript (dictionary-like access redirected to the underlying backing dict), performs the query needed to retrieve just that value. Example name, following the overall naming pattern: find_scalar()
This may be more natural than pre-defining an iterable of fields to retrieve, then unpacking a resulting tuple of those retrieved values. (Sometimes you don't want your data cached application-side… but still want the benefit of those descriptor objects.)
Subj, I'm getting error like
Q(SomeType, 'some_field', Array('some_field', kind=Reference('', kind='some_class'))) does not allow $ne comparison.
Define a Markdown field for storage of Markdown formatted text which can self-render using the MarkupSafe __html__
protocol when embedded in a template for web-safe rendering.
The hierarchical branch has diverged substantially and incorporated otherwise unrelated fixes, improvements, and cleanup. Extract these cleanup items for 1.1.2 release. See: #41
The current version (5.4.1) as suggested in travis-ci/travis-ci#4756 is quite dated at this point. Problems have been encountered (relating to ABC metaclass usage) which may be resolved in subsequent (supported) releases. Depends on travis-ci/travis-ci#7507 upstream.
There is some hope, for example, utilizing nightly builds as per the recommendation of njs
from #pypy
on Freenode, ref:
During construction of a PluginReferenceField
if the reference may not be explicit, and no choices
value has been passed in, automatically populate choices
from available plugin names.
This would require duplication of the "is explicit or not" check which should be moved to a private @property
.
Following the example of the GeoJSON support code.
Example usage:
class Thing(Lockable):
__collection__ = 'things'
lockable_instance = Thing()
lockable_instance.insert_one()
try:
with lockable_instance:
... # Critical section.
except lockable_instance.Locked:
... # Failure scenario.
User-contributed report from @djdduty, recent installation without other dependency on PyMongo, but with a dependency on Marrow Mongo, installed a version with new, breaking changes. Adapt to the change, make the import (and support) conditional, and/or pin the maximum version. Each is a progressively simpler step towards mitigation, with the last the least effective and complete.
Patch mitigation for users of Marrow Mongo, explicitly pin PyMongo's version in your own dependencies:
pymongo>=3.2,<4.0
Ref: Breaking Changes in 4.0 (official changelog)
<djdduty> Confirmed that results in pymongo 3.12.3 in CI
Data sanitization is often a PITA. This is a minor detail that can save a lot of time and repeated boilerplate elsewhere.
The addition of a Mapping
field (a specialization of Array
) to allow retrieval and storage of an ordered mapping of objects utilizing a common key field.
Specifically, the following BSON document:
{things: [{name: 'bob', age: 22}, {name: 'alice', potato: true}]}
Would be retrievable:
class Thing(Document):
class Person(Document):
name = String()
age = Integer()
potato = Boolean()
things = Mapping('.Person')
Thing.find_one().things['bob'].age == 22
Work has already begun on this feature:
Im using Flask_Script to run Flask on a localserver.
I am unclear what is causing the error below.
error trace when run the app though manager
if __name__ == '__main__':
manager.run()
Traceback (most recent call last):
File "/pathto/myapp/manage.py", line 154, in <module>
manager.run()
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/commands.py", line 425, in __call__
**self.server_options)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask/app.py", line 841, in run
run_simple(host, port, self, **options)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/serving.py", line 706, in run_simple
reloader_type)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 263, in run_with_reloader
reloader.run()
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 144, in run
self.extra_files):
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 22, in _iter_module_files
filename = getattr(module, '__file__', None)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/marrow/mongo/util/__init__.py", line 62, in __getattr__
self.__dict__[name] = load(name, self._namespace)
File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/marrow/package/loader.py", line 81, in load
raise LookupError('Unknown plugin "' + target + '"; found: ' + ', '.join(allowable))
LookupError: Unknown plugin "__file__"; found: Polygon, Point, GeoJSONCoord, GeometryCollection, Document, GeoJSON, MultiPoint, MultiLineString, LineString, MultiPolygon
I can run unit tests fine (eg see below) indicating the issue is related to code execution during the flask_script manager.runserver() command
def test_password_setter(self):
user_role = Permission.USER
u = User(email='[email protected]', role=user_role)
u.password='password'
self.assertTrue(u.password_hash is not None)
marrow User object
from marrow.mongo import Index
#here is where the error is initiated when running thouth manage.runserver()
from marrow.mongo.field import Array, Boolean, Date, Integer, Number, ObjectId, Reference, String, Timestamp
from marrow.mongo.trait import Queryable
class User(Queryable, UserMixin):
__collection__ = 'user'
confirmed = Boolean(default=False)
first_name = String()
last_name = String()
email = String()
password_hash = String()
#using reference makes updates and queries too complicated
role = String(choices=[Permission.USER, Permission.MANAGER, Permission.ADMINISTRATOR], default=Permission.USER)
created = Timestamp()
revised_on = Date()
_email = Index('email', unique=True)
def __init__(self, **kwargs):
super(User, self).__init__(**kwargs)
if self.role is None:
if self.email == current_app.config['ADMIN_EMAIL']:
role = Permission.ADMINISTRATOR
if self.role is None:
#default
self.role = Permission.USER
@password.setter
def password(self, password):
self.password_hash = generate_password_hash(password)
def _create_collection_and_drop():
assert mongo.db is not None
#beware, will remove all data from collections
db = mongo.db
User.bind(db)
User.create_collection(drop=True)
As a test, Flask starts fine when I change the loader.load() method to:
if namespace and ':' not in target:
allowable = dict((i.name, i) for i in iter_entry_points(namespace))
if target not in allowable:
if target == '__file__':
return None
else:
raise LookupError('Unknown plugin "' + target + '"; found: ' + ', '.join(allowable))
It can be handy during transitional times to be able to point old import references at new locations on access. This can help prevent bit rot, or allow for read compatibility with other codebases.
Currently there is some either unacceptable behaviour (only Registry
attributes are listed in completions) or erroneous behaviour (exceptions in REPL shells upon typing .
and triggering completion). Depending on the IDE or editor's mechanism of collecting identifiers, this will also prohibit their own symbol indexing.
Attempting tab completion (or equivalent) or performing introspection (via dir()
and friends) should list the available plugins, accounting for optional requirements.
Derive a new Field or Index instance from an existing one. E.g.:
from marrow.mongo import Document, Field
class Base(Document):
field = Field()
class Child(Base):
field = Base.field.adapt(default="Hi.")
Traits:
Collection — "Active Collection" features for management, such as collection and index creation; minimal, encouraging use of get_collection()
and PyMongo methods where possible.
Queryable — "Active Collection" bridging to "Active Record" where such proxying serves the purpose of translation from simplified Marrow Mongo concepts (such as parametric interpretation) and is demonstrably useful. Less minimal.
Discuss.
Document is a member of an acyclic directed graph, or tree, with slugs and paths. Order preservation is not implied beyond parent/child and ancestor/descendant relationships.
Subj, I'm trying to implement mixin kind-of
class Complainable(object):
complainer = ReferenceField('...')
And use it like
class SomeObj(Complainable, Document):
...
Then I try to create query object:
SomeObj.complainer == ObjectId()
I'll get error 'Reference' object has no attribute '__name__'
Given the following test case:
from marrow.mongo import Document, Index
from marrow.mongo.field import String
class Base(Document):
foo = String()
_foo = Index('foo')
class Sub(Base):
bar = String()
_bar = Index('bar')
Indexes provided by this class and all subclasses should be made available as an attribute, and calls to create_collection
or create_indexes
should accept an additional argument (defaulting to True
) to create subclass indexes in addition to the primary class indexes.
Ensure headings are always sans-serif, blockquotes are always serif.
Anywhere sans-serif is utilized prefer the macOS system font. Select appropriate weights to maximize use of this font and ensure ligatures are enabled.
Utilize a web-safe version of Monofur for fixed-width fonts.
Limit maximal text width to something like 40em
to maximize readability; this will have interactions with the variable column layout.
Matches other use, such as in the Contributing chapter.
Link to internal documentation, PyMongo, or Python standard library documentation as appropriate.
Especially useful for tag sets.
Deprecate the numeric indexing manipulation scheme for "very large positive value" use to prevent a field from being filled positionally. Something like the following in Document.__init__
(and subsequent field attribute definition) should do:
fields = iter(i for i in self.__fields__.items() if isinstance(i, Field) and i.positional)
# We translate positional to keyword arguments ourselves to facilitate per-field inclusion.
# Also to correct for accidental inclusion of Attributes instances, etc.
for arg in args:
for name, field in fields:
if name in kw:
raise TypeError("Positional value overridden by keyword argument: " + name)
kw[name] = arg
break
super(Document, self).__init__(**kw)
...
positional = Attribute(default=True) # If True, will be accepted positionally.
Hi,
I came across your project a few days ago, looking for some alternatives to an in-house mongo-wrapper and schema-definition utility.
I was super impressed with marrow-mongo, and the entire echo-system, and I've been trying to use it in my project.
I'm trying to define a document structure with some required fields, and with type validation on each field. Unfortunately, I was not able to get any of these validations to work out of the box (see ipython snippet).
In [27]: class A(Document):
...: n = Number(required=True, validator=never)
...:In [28]: A()
Out[28]: A()In [29]: A(n=1)
Out[29]: A(n=1)
It looks as though the required
attribute is completely ignored, and so does the validator attribute (however, I was able to enforce validation by adding the Validated
mixin to the Field
class, but this will force me to redefine each field type I wish to use).
Looking at the code, I wasn't able to find anywhere where these validations are enforced by default, but I'm probably missing something. I'd appreciate you help.
Thanks!
from_dict
† and to_dict
imperative methods.as_dict
‡ getter.This would be the JSON representation in plain dictionary form.
from_json
† and to_json
imperative methods.as_json
‡ getter.Implementable as a json.dumps(self.as_dict)
from_hhc
† and to_hhc
imperative methods.as_hhc
‡ getter.For a comparison of the encoding efficiencies involved, given the sample value ObjectID('5e005bc99d79eb6bd5b75096')
:
Binary: 12 bytes. (Might not look smaller, but it is.)
b'^\x00[\xc9\x9dy\xebk\xd5\xb7P\x96'
Raw BSON: 15 bytes minimum. 1 identification byte + null-terminated key name + 12 byte compact identifier struct.
HHC: 16 bytes.
'CoaE_HfG_uPXUI_d'
Base64: 16 bytes. This example is lucky, there are no URL-unsafe characters like +
being generated.
'XgBbyZ1562vVt1CW'
Hexadecimal: 24 bytes.
'5e005bc99d79eb6bd5b75096'
JSON: 36 bytes as a hex-encoded string within a compound object with type-identifying key name.
'{"$oid": "5e005bc99d79eb6bd5b75096"}'
† Class method.
‡ On-access property.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.