marrow / mongo Goto Github PK

Derive a new Field or Index instance from an existing one. E.g.:

from marrow.mongo import Document, Field

class Base(Document):
    field = Field()

class Child(Base):
    field = Base.field.adapt(default="Hi.")

The current version (5.4.1) as suggested in travis-ci/travis-ci#4756 is quite dated at this point. Problems have been encountered (relating to ABC metaclass usage) which may be resolved in subsequent (supported) releases. Depends on travis-ci/travis-ci#7507 upstream.

There is some hope, for example, utilizing nightly builds as per the recommendation of njs from #pypy on Freenode, ref:

• https://github.com/python-trio/trio/blob/master/.travis.yml
• https://github.com/python-trio/trio/blob/master/ci/travis.sh#L20

Document maintains state described by a cyclic directed graph of allowable string tag-based state transitions.

Currently there is some either unacceptable behaviour (only Registry attributes are listed in completions) or erroneous behaviour (exceptions in REPL shells upon typing . and triggering completion). Depending on the IDE or editor's mechanism of collecting identifiers, this will also prohibit their own symbol indexing.

Attempting tab completion (or equivalent) or performing introspection (via dir() and friends) should list the available plugins, accounting for optional requirements.

On this page: https://mongo.webcore.io/installation.html

You have a link towards the bottom for the contributing guide. If clicked, this attempts to take you to https://mongo.webcore.io/CONTRIBUTING.md which appears to hang indefinitely by the spinning icon at the top of the page. In the javascript console, you get
GET https://mongo.webcore.io/CONTRIBUTING.md 404 ()

Serializations

Dictionary

• from_dict† and to_dict imperative methods.
• as_dict‡ getter.

This would be the JSON representation in plain dictionary form.

JSON

• from_json† and to_json imperative methods.
• as_json‡ getter.

Implementable as a json.dumps(self.as_dict)

HHC

https://pypi.org/project/hhc/

• from_hhc† and to_hhc imperative methods.
• as_hhc‡ getter.

Comparison

For a comparison of the encoding efficiencies involved, given the sample value ObjectID('5e005bc99d79eb6bd5b75096'):

• Binary: 12 bytes. (Might not look smaller, but it is.)
  b'^\x00[\xc9\x9dy\xebk\xd5\xb7P\x96'

• Raw BSON: 15 bytes minimum. 1 identification byte + null-terminated key name + 12 byte compact identifier struct.

• HHC: 16 bytes.
  'CoaE_HfG_uPXUI_d'

• Base64: 16 bytes. This example is lucky, there are no URL-unsafe characters like + being generated.
  'XgBbyZ1562vVt1CW'

• Hexadecimal: 24 bytes.
  '5e005bc99d79eb6bd5b75096'

• JSON: 36 bytes as a hex-encoded string within a compound object with type-identifying key name.
  '{"$oid": "5e005bc99d79eb6bd5b75096"}'

† Class method.
‡ On-access property.

Following the example of the GeoJSON support code.

Add the following classmethods to the Document class:

• create_collection
• get_collection
• create_indexes

Subj, I'm getting error like

Q(SomeType, 'some_field', Array('some_field', kind=Reference('', kind='some_class'))) does not allow $ne comparison.

The addition of a Mapping field (a specialization of Array) to allow retrieval and storage of an ordered mapping of objects utilizing a common key field.

Specifically, the following BSON document:

{things: [{name: 'bob', age: 22}, {name: 'alice', potato: true}]}

Would be retrievable:

class Thing(Document):
    class Person(Document):
        name = String()
        age = Integer()
        potato = Boolean()
    
    things = Mapping('.Person')

Thing.find_one().things['bob'].age == 22

Work has already begun on this feature:

• b7c64ec Added mapping field and update localized to utilize it.

User-contributed report from @djdduty, recent installation without other dependency on PyMongo, but with a dependency on Marrow Mongo, installed a version with new, breaking changes. Adapt to the change, make the import (and support) conditional, and/or pin the maximum version. Each is a progressively simpler step towards mitigation, with the last the least effective and complete.

Patch mitigation for users of Marrow Mongo, explicitly pin PyMongo's version in your own dependencies:

pymongo>=3.2,<4.0

Ref: Breaking Changes in 4.0 (official changelog)

<djdduty> Confirmed that results in pymongo 3.12.3 in CI

As an extension to Alias fields (#12), the ability to mark an attribute as a deprecated alias can be helpful when migrating older code.

• be4687b Added deprecation capability.

Hi,
I came across your project a few days ago, looking for some alternatives to an in-house mongo-wrapper and schema-definition utility.
I was super impressed with marrow-mongo, and the entire echo-system, and I've been trying to use it in my project.

I'm trying to define a document structure with some required fields, and with type validation on each field. Unfortunately, I was not able to get any of these validations to work out of the box (see ipython snippet).

In [27]: class A(Document):
...: n = Number(required=True, validator=never)
...:

In [28]: A()
Out[28]: A()

In [29]: A(n=1)
Out[29]: A(n=1)

It looks as though the required attribute is completely ignored, and so does the validator attribute (however, I was able to enforce validation by adding the Validated mixin to the Field class, but this will force me to redefine each field type I wish to use).

Looking at the code, I wasn't able to find anywhere where these validations are enforced by default, but I'm probably missing something. I'd appreciate you help.

Thanks!

Given they have an _id key.

During construction of a PluginReferenceField if the reference may not be explicit, and no choices value has been passed in, automatically populate choices from available plugin names.

This would require duplication of the "is explicit or not" check which should be moved to a private @property.

Given the following test case:

from marrow.mongo import Document, Index
from marrow.mongo.field import String

class Base(Document):
    foo = String()
    _foo = Index('foo')

class Sub(Base):
    bar = String()
    _bar = Index('bar')

Indexes provided by this class and all subclasses should be made available as an attribute, and calls to create_collection or create_indexes should accept an additional argument (defaulting to True) to create subclass indexes in addition to the primary class indexes.

Subj, I'm trying to implement mixin kind-of

class Complainable(object):
    complainer = ReferenceField('...')

And use it like

class SomeObj(Complainable, Document):
    ...

Then I try to create query object:

SomeObj.complainer == ObjectId()

I'll get error 'Reference' object has no attribute '__name__'

Getting this idea out of my brain onto "paper" while it's fresh, permit the return of scalar attribute proxies from find_one(), that is, an apparent instance of the targeted Document sub–class that upon access to an attribute (field descriptor) or subscript (dictionary-like access redirected to the underlying backing dict), performs the query needed to retrieve just that value. Example name, following the overall naming pattern: find_scalar()

This may be more natural than pre-defining an iterable of fields to retrieve, then unpacking a resulting tuple of those retrieved values. (Sometimes you don't want your data cached application-side… but still want the benefit of those descriptor objects.)

• Explicit interpretation of naive datetime timezone; default to UTC, allow local alias (identify server-local timezone), 'naive' alias (strip timezone), timezone name, explicit tzinfo instance, or use of callback returning one of these.
• Explicit Python-local timezone interpretation; defaulting to pass-through, allowances as above.

Existing work in c0cb7d2

Especially useful for tag sets.

It can be handy during transitional times to be able to point old import references at new locations on access. This can help prevent bit rot, or allow for read compatibility with other codebases.

Define a Markdown field for storage of Markdown formatted text which can self-render using the MarkupSafe __html__ protocol when embedded in a template for web-safe rendering.

Matches other use, such as in the Contributing chapter.

We can begin the search from the line pulling in the record (.find_one, &c.) through all reachable points from there, looking for all attribute and dictionary-like access to that object.

¹ At runtime.

Im using Flask_Script to run Flask on a localserver.

I am unclear what is causing the error below.

error trace when run the app though manager

if __name__ == '__main__':
    manager.run()

Traceback (most recent call last):
  File "/pathto/myapp/manage.py", line 154, in <module>
    manager.run()
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask_script/commands.py", line 425, in __call__
    **self.server_options)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/flask/app.py", line 841, in run
    run_simple(host, port, self, **options)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/serving.py", line 706, in run_simple
    reloader_type)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 263, in run_with_reloader
    reloader.run()
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 144, in run
    self.extra_files):
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/werkzeug/_reloader.py", line 22, in _iter_module_files
    filename = getattr(module, '__file__', None)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/marrow/mongo/util/__init__.py", line 62, in __getattr__
    self.__dict__[name] = load(name, self._namespace)
  File "/home/user/.virtualenvs/myapp/lib/python3.5/site-packages/marrow/package/loader.py", line 81, in load
    raise LookupError('Unknown plugin "' + target + '"; found: ' + ', '.join(allowable))
LookupError: Unknown plugin "__file__"; found: Polygon, Point, GeoJSONCoord, GeometryCollection, Document, GeoJSON, MultiPoint, MultiLineString, LineString, MultiPolygon

I can run unit tests fine (eg see below) indicating the issue is related to code execution during the flask_script manager.runserver() command

    def test_password_setter(self):
        user_role = Permission.USER
        u = User(email='[email protected]', role=user_role)
        u.password='password'
        self.assertTrue(u.password_hash is not None)

marrow User object

from marrow.mongo import Index
#here is where the error is initiated when running thouth manage.runserver()
from marrow.mongo.field import Array, Boolean, Date, Integer, Number, ObjectId, Reference, String, Timestamp 
from marrow.mongo.trait import Queryable

class User(Queryable, UserMixin):
    __collection__ = 'user'
    confirmed = Boolean(default=False)
    first_name = String()
    last_name = String()
    email = String()
    password_hash = String()
    #using reference makes updates and queries too complicated
    role = String(choices=[Permission.USER, Permission.MANAGER, Permission.ADMINISTRATOR], default=Permission.USER)
    created = Timestamp()
    revised_on = Date()
    _email = Index('email', unique=True)

    def __init__(self, **kwargs):
        super(User, self).__init__(**kwargs)
        if self.role is None:
            if self.email == current_app.config['ADMIN_EMAIL']:
                role = Permission.ADMINISTRATOR
            if self.role is None:
                #default
                self.role = Permission.USER

    @password.setter
    def password(self, password):
        self.password_hash = generate_password_hash(password)

def _create_collection_and_drop():
    assert mongo.db is not None
    #beware, will remove all data from collections
    db = mongo.db
    User.bind(db)
    User.create_collection(drop=True)

As a test, Flask starts fine when I change the loader.load() method to:

	if namespace and ':' not in target:
		allowable = dict((i.name,  i) for i in iter_entry_points(namespace))
		if target not in allowable:
			if target == '__file__':
				return None
			else:
				raise LookupError('Unknown plugin "' + target + '"; found: ' + ', '.join(allowable))

A String field specialization utilizing pathlib.PurePosixPath conversion.

Example usage:

class Thing(Lockable):
    __collection__ = 'things'

lockable_instance = Thing()
lockable_instance.insert_one()

try:
    with lockable_instance:
        ... # Critical section.

except lockable_instance.Locked:
    ... # Failure scenario.

• 156e9bc Added initial Lockable trait; untested.
• 3218f47 Lockable tweaks and initial tests.
• 8a462fc Full functional lock coverage.
• b2d3eaa Locks can now be waited upon.
• ad7d4f9 Armour against slow Lockable tests and tweak in re-locking behaviour.

Data sanitization is often a PITA. This is a minor detail that can save a lot of time and repeated boilerplate elsewhere.

Link to internal documentation, PyMongo, or Python standard library documentation as appropriate.

Traits:

• Collection — "Active Collection" features for management, such as collection and index creation; minimal, encouraging use of get_collection() and PyMongo methods where possible.

• Queryable — "Active Collection" bridging to "Active Record" where such proxying serves the purpose of translation from simplified Marrow Mongo concepts (such as parametric interpretation) and is demonstrably useful. Less minimal.

Discuss.

The current extension in the "shop" fails badly; possibly this is built-in now? Should jump to the diff for that revision.

Ensure headings are always sans-serif, blockquotes are always serif.

Anywhere sans-serif is utilized prefer the macOS system font. Select appropriate weights to maximize use of this font and ensure ligatures are enabled.

Utilize a web-safe version of Monofur for fixed-width fonts.

Limit maximal text width to something like 40em to maximize readability; this will have interactions with the variable column layout.

I appear to have a case where Pypy (5.4.1) does not interpret a MutableMapping.register()'d class as a mapping as per CPython 2/3 on the same codebase as seen by PyMongo. (A… complex case.)

Usage of manual ABC registration is the primary issue; avoiding the indirection of the wrapped objects avoids the symptom in follow-up testing. Subclassing the ABC does work correctly in this particular instance.

This does not correct the overall issue with manual registration, however, as I have classes with incompatible metaclasses that I also wish to participate in the ABC. (With prior Travis failures and work-arounds for the previously encountered incidents of this.). However, Ops is a biggie.

The hierarchical branch has diverged substantially and incorporated otherwise unrelated fixes, improvements, and cleanup. Extract these cleanup items for 1.1.2 release. See: #41

Hey, hi!
I really liked the Marrow Mongo approach.

We have been bitten several times by mongoengine and were considering migrating to something else...

• Is Marrow Mongo used on production? By many people/companies? In which use cases?
• How confident are you of it correctness, currently?
• How was your experience migrating from Mongoengine?

I just want to learn more about the project and see if is a fit for us. I am sorry for the trouble!

Thanks and congratulations!

Document is a member of an acyclic directed graph, or tree, with slugs and paths. Order preservation is not implied beyond parent/child and ancestor/descendant relationships.

All URI values, absolute or relative, for any protocol, with optional protocol restriction. Some code already committed, ref:

• bd059f5 Link field to store absolute and relative URI, including URL and URN.
• cba8a78 Link field simplification.

Deprecate the numeric indexing manipulation scheme for "very large positive value" use to prevent a field from being filled positionally. Something like the following in Document.__init__ (and subsequent field attribute definition) should do:

fields = iter(i for i in self.__fields__.items() if isinstance(i, Field) and i.positional)

# We translate positional to keyword arguments ourselves to facilitate per-field inclusion.
# Also to correct for accidental inclusion of Attributes instances, etc.
for arg in args:
    for name, field in fields:
        if name in kw:
            raise TypeError("Positional value overridden by keyword argument: " + name)

        kw[name] = arg
        break

super(Document, self).__init__(**kw)

...

positional = Attribute(default=True)  # If True, will be accepted positionally.