Coder Social home page Coder Social logo

axiom's Introduction

Twisted

gitter_ rtd_ pypi_ ci_

For information on changes in this release, see the NEWS file.

What is this?

Twisted is an event-based framework for internet applications, supporting Python 3.6+. It includes modules for many different purposes, including the following:

  • twisted.web: HTTP clients and servers, HTML templating, and a WSGI server
  • twisted.conch: SSHv2 and Telnet clients and servers and terminal emulators
  • twisted.words: Clients and servers for IRC, XMPP, and other IM protocols
  • twisted.mail: IMAPv4, POP3, SMTP clients and servers
  • twisted.positioning: Tools for communicating with NMEA-compatible GPS receivers
  • twisted.names: DNS client and tools for making your own DNS servers
  • twisted.trial: A unit testing framework that integrates well with Twisted-based code.

Twisted supports all major system event loops -- select (all platforms), poll (most POSIX platforms), epoll (Linux), kqueue (FreeBSD, macOS), IOCP (Windows), and various GUI event loops (GTK+2/3, Qt, wxWidgets). Third-party reactors can plug into Twisted, and provide support for additional event loops.

Installing

To install the latest version of Twisted using pip:

$ pip install twisted

Additional instructions for installing this software are in the installation instructions.

Documentation and Support

Twisted's documentation is available from the Twisted Matrix website. This documentation contains how-tos, code examples, and an API reference.

Help is also available on the Twisted mailing list.

There is also an IRC channel, #twisted, on the Libera.Chat network. A web client is available at web.libera.chat.

Unit Tests

Twisted has a comprehensive test suite, which can be run by tox:

$ tox -l                       # to view all test environments
$ tox -e nocov                 # to run all the tests without coverage
$ tox -e withcov               # to run all the tests with coverage
$ tox -e alldeps-withcov-posix # install all dependencies, run tests with coverage on POSIX platform

You can test running the test suite under the different reactors with the TWISTED_REACTOR environment variable:

$ env TWISTED_REACTOR=epoll tox -e alldeps-withcov-posix

Some of these tests may fail if you:

  • don't have the dependencies required for a particular subsystem installed,
  • have a firewall blocking some ports (or things like Multicast, which Linux NAT has shown itself to do), or
  • run them as root.

Static Code Checkers

You can ensure that code complies to Twisted coding standards:

$ tox -e lint   # run pre-commit to check coding stanards
$ tox -e mypy   # run MyPy static type checker to check for type errors

Or, for speed, use pre-commit directly:

$ pipx run pre-commit run

Copyright

All of the code in this distribution is Copyright (c) 2001-2024 Twisted Matrix Laboratories.

Twisted is made available under the MIT license. The included LICENSE file describes this in detail.

Warranty

THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE USE OF THIS SOFTWARE IS WITH YOU.

IN NO EVENT WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY, BE LIABLE TO YOU FOR ANY DAMAGES, EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Again, see the included LICENSE file for specific legal details.

axiom's People

Contributors

ddormer avatar dependabot[bot] avatar exarkun avatar glyph avatar hawkowl avatar jsza avatar ldanielburr avatar lvh avatar mithrandi avatar palfrey avatar techtonik avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

mithrandi glyph hawkowl pombredanne ldanielburr ddormer perkinslr linkedblogs giorgil opacam dalavancloud jsza palfrey hammer2900 stjordanis python-repository-hub isabella232 bryanchance

axiom's Issues

Tests rely on refcounting / GC semantics for Axiom type definitions

It appears that the only real reason the asserted behaviour is necessary is for testing purposes; supporting this in application code is undesirable (and the application code is unlikely to function correctly for other reasons). So instead, we adjust the one test that relies on this to explicitly remove the schema from the cache, and remove the other tests.

Imported from Launchpad using lp2gh.

Failing test case axiom.test.test_xatop.ProcessConcurrencyTestCase.testNewItemTypeInSubprocess

I have no idea what this means, but this test case fails on trunk for me.

Axiom ❯ trial axiom.test.test_xatop.ProcessConcurrencyTestCase.testNewItemTypeInSubprocess ⏎
axiom.test.test_xatop
ProcessConcurrencyTestCase
testNewItemTypeInSubprocess ... [ERROR]

[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessDone: A process has ended without apparent errors: process finished with exit code 0.

axiom.test.test_xatop.ProcessConcurrencyTestCase.testNewItemTypeInSubprocess

Ran 1 tests in 0.249s

FAILED (errors=1)

Imported from Launchpad using lp2gh.

The batch process is not started up automatically

This was filed in the old Divmod issue tracker as ticket 2963.

When axiomatic starts up, it does nothing special with the batch processing system. If application code explicitly requests something from the remote batch processor, the batch process will get started up. However, if there are pending items in need of remote processing, these will not cause the batch process to start (so the items are not processed). And the creation of new items for remote processing is also not sufficient to start the process.

This is hard to fix for real, but an improvement is to at least start the batch process when new items for remote processing are created.

Imported from Launchpad using lp2gh.

Get rid of PYTHONPATH mangling

A few pieces of code that launch a child process try to build a PYTHONPATH environment variable by stitching sys.path together; this produces incorrect results, depending on what happens to be on sys.path. It is also unnecessary: if the current value of PYTHONPATH resulted in the current value of sys.path, then it should have exactly the same effect in the child process without any modification.

Imported from Launchpad using lp2gh.

Switch to versioneer

Versioneer is already used in a number of Twisted-related projects (eg. Nevow, Imaginary), we should probably follow suit. (Also it beats managing stuff manually.)

Decorator for installables

Something like this:

@empowers(IFoo)
class MyPowerup(Item):
    …

which would be equivalent to:

@implementer(IFoo)
class MyPowerup(Item):
    powerupInterfaces = [IFoo]

The main motivation for this is that it makes no sense to have an interface in powerupInterfaces that you do not implement. Note that you can still implement interfaces that you don't power up for:

@implementer(INotPowerupInterface)
@empowers(IFoo)
class MyPowerup(Item):
    …

works fine.

cache semantics are wrong on PyPy

I have not managed to reproduce this in isolation outside the context of twisted/imaginary#64 just yet, but I believe I'm seeing Item instances with the same storeID for the same store existing at the same time.

Superficially it appears that the weakref to an object returns None while there are still strong references held elsewhere in the program. I feel like I must be wrong about that though.

`axiom.batch.processor` mishandles weak references

axiom.batch has a sequence of code that looks like this:

_processors = weakref.WeakValueDictionary()
# ...
def processor(forType):
# ...
    if forType not in _processors:
    # ...
        _processors[forType] = #...
    return _processors[forType]

In the case where the processor is not already in the WeakValueDictionary, there will be no reference to the processor anywhere, and thus it may be garbage collected before the final return statement, resulting in a KeyError trying to get it out of the dictionary again.

Broken upgrader in `SwordUpgradeTest`

The failure is in axiom.test.test_upgrading.SwordUpgradeTest.testUpgradeWithMissingVersion, but it only occurs when a certain dict ordering happens. PYTHONHASHSEED=1290646218 produces one such ordering; this wasn't noticed before because the default seed doesn't produce the ordering that causes the failure.

First, the failing test traceback:

[ERROR]
Traceback (most recent call last):
  File "/home/mithrandi/code/axiom/axiom/test/test_upgrading.py", line 189, in afterUpgrade
    self._testPlayerAndSwordState(player, sword)
  File "/home/mithrandi/code/axiom/axiom/test/test_upgrading.py", line 368, in _testPlayerAndSwordState
    assert not sword.__legacy__
exceptions.AssertionError: 

axiom.test.test_upgrading.SwordUpgradeTest.testUpgradeWithMissingVersion

The assertion fires because the sword item has not been fully upgraded. The reason for this is that the player1to2 upgrader does something that confuses the upgrade system. When the store is opened, Axiom looks at all of the currently-existing type versions in the store; any version that is out of date is handed to the upgrade manager in order to be upgraded. In the case of this test, the versions are (Player, 1) and (Sword, 1). The upgrade manager loops over the types to be upgraded; for each type, it queries for all items of that type, and fully upgrades them (running upgraders until the current schema version is reached), before moving on to the next type/version.

The player1to2 upgrader calls upgradePlayerAndSword, which retrieves both the sword and the player, and upgrades both of them to version 2. If this upgrader runs first, then the sword item will be at version 2. However, recall that the entry in the upgrader's list is (Sword, 1); when it queries for items of this type, it obviously finds none. However, (Sword, 2) is not in its list, so the sword version 2 item is never upgraded to sword version 3. (The inverse problem does not occur when sword1to2 runs first, because there is no Player v3 type).

How to proceed? I'm not sure if what this upgrader is doing should just be considered unsupported, and this upgrader (and corresponding part of the test) should be removed, or if we should find some way to make this functionality actually work (probably not impossible, but I don't see an easy way to make it work at the moment, so likely quite a lot of work).

Helper for declaring legacy items

Constructing a legacy declaration for a current item version by hand is not something people should be doing. There's a script floating around to do this, we should ship it (in a high quality, properly tested form) with Axiom instead of handing down the secret knowledge from generation to generation.

path(relative=False) doesn't actually do anything

The parameter has a default value of True, but in fact is entirely ignored: the attribute it looks like it should control is always set to True by __init__ regardless of the value of this parameter.

Store API for setting the journal mode

My idea is to add a parameter to Store (and maybe make it default to wal, since I think that'll be a better default for most users? Although you could view this as a compatibility issue), and stick it in an instance attribute so that SubStore can inherit it from the parent store. There should also maybe be an axiomatic flag to select this?

reference `reftype` should be enforced

reftype doesn't do the most obvious thing it could do:

from axiom.store import Store
from axiom.item import Item
from axiom.attributes import reference

class X(Item):
foo = reference()

class Y(Item):
bar = reference(reftype=X)

class Z(Item):
baz = reference()

def main():
s = Store()
print Y(store=s, bar=Z(store=s)).bar

if name == 'main':
main()

Instead, I guess it does other things - like serve as documentation about what you should be putting here. Also it provides a deletion optimization (which ends up being incorrect if you disregard the documentation purpose and make the reference to a different type ...)

It seems Axiom should not allow assignment of the wrong type.

Imported from Launchpad using lp2gh.

Do some better benchmarking stuff

There are currently two different types of benchmark, epsilon.benchmark-based in axiom.benchmark and ad-hoc in benchmarks/, and both of them kinda suck.

Axiom should use stdlib sqlite3 on PyPy 2.0 and beyond

PyPy2.0 now contains cffi as a standard library addition, and also has a sqlite3 module that uses cffi. cffi sqlite3 is better than pysqlite2, but that code has not made it back into pysqlite2.

It seems that pysqlite2 changes so rarely these days it might be a good idea to use stdlib sqlite3 preferentially over pysqlite2; it'll be an easier patch, at least :)

Imported from Launchpad using lp2gh.

Potential data loss involving reference attributes and upgraders

This is another PyPy-mostly issue; I believe it will be possible to reproduce on CPython, but the code required to do so is fairly pathological so I don't expect anyone to have run into it there.

  • Start with creating an item (at the latest version), with a reference attribute (call this item type R) pointing at a legacy item (for example, the item being upgraded, in an upgrader that is not to the latest version).
  • The newly-created Item object will have a reference to the legacy item DummyItem object in the axiom_memory_<reference attribute name> attribute.
  • When the legacy item is upgraded, Item.committed will set the __store attribute to None.
  • If a strong reference to the R item is kept (or there is no strong reference, but the item is fetched out of the cache before the object is GCed), reference.__get__ will get the old DummyItem object from the axiom_memory_<name> attribute. It will then test this by calling _currentlyValidAsReferentFor on it, which will return False due to store now being None; this, in turn, will make __get__ return None.

Once the R item is loaded from the database again, everything will be fine; however, data loss may occur as a result of the reference attribute returning None in the meantime. There is a particularly nasty interaction with powerups here; if _PowerupConnector.powerup appears as None during a call to powerupsFor(), the powerup connector will be deleted! On the plus side, it is impossible to get a reference to the item created in powerUp(), so I think triggering this particular case in CPython is impossible. In fact, in general, writing code in an upgrader that retains a strong reference to an item outside of the upgrader requires some fairly pathological code, so this is also unlikely.

However, on PyPy, GC semantics mean that such item objects may stay alive for much longer, and thus if they are loaded from the database again, they will be loaded from the item cache, without a strong reference to them being needed.

I think the fix for this is to avoid setting __store to None in Item.committed for legacy items. This is a mere change in indentation, so it is possible that this bug is actually just an error in indentation.

stop storing plaintext passwords

It's really unacceptable that Axiom stores passwords in plain text. We should hash them according to best practices (i.e. let's not home-brew some nonsense for obscuring them, let's use a legit PBKDF).

https://github.com/lvh/txscrypt has a pretty great API for this, but it does introduce a Deferred where we didn't have one before.

Deprecate startswith/endswith on bytes columns

These are implemented using the LIKE operator, which does not work on BLOB values when SQLite 3 is compiled with the SQLITE_LIKE_DOESNT_MATCH_BLOBS option (which Debian is now doing). The SQLite documentation has this to say:

SQLITE_LIKE_DOESNT_MATCH_BLOBS

This compile-time option causes the LIKE operator to always return False if either operand is a BLOB. The default behavior of LIKE is that BLOB operands are cast to TEXT before the comparison is done.

This compile-time option makes SQLite run more efficiently when processing queries that use the LIKE operator, at the expense of breaking backwards compatibility. However, the backwards compatibility break may be only a technicality. There was a long-standing bug in the LIKE processing logic (see https://www.sqlite.org/src/info/05f43be8fdda9f) that caused it to misbehavior for BLOB operands and nobody observed that bug in nearly 10 years of active use. So for more users, it is probably safe to enable this compile-time option and thereby save a little CPU time on LIKE queries.

Thus I would suggest deprecating this functionality and removing any reliance on it from the test suite; this would allow any users with a suitably-compiled SQLite 3 to continue using the functionality for the time being.

Default log rotation is bad

The default logging settings for axiomatic.log (as well as Mantissa's http.log, see twisted/mantissa#63) are unbounded, and use ordinal rotation; this means that rotation cost scales linearly with the number of historic log files, and this number is unbounded, resulting in your process eventually blocking for minutes at a time renaming tens of thousands of files every time a new log file is started. This is not ideal, and also not obvious to debug.

SubStore.createNew is too easy to misuse

createNew takes a list/tuple of path segments; if you pass a string instead, then each letter of the string ends up used as a path component. While this is relatively harmless, it is also unlikely to be the desired behaviour; and in fact, the tests already make this mistake in one place.

Imported from Launchpad using lp2gh.

setup.py doesn't install misc files

Installing Axiom with the new setup.py misses a bunch of files that are not Python modules, such as the .tbz historical databases, and batch.tac. These need to be listed in MANIFEST.in so that they actually get installed.

Imported from Launchpad using lp2gh.

Intermittent failure in reactor selection tests

Looks like:

===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessDone: A process has ended without apparent errors: process finished with exit code 0.
axiom.test.test_axiomatic.StartTests.test_reactorSelectionShortOptionStyle
-------------------------------------------------------------------------------

See, for example, https://travis-ci.org/twisted/axiom/jobs/113794614.

Absolutely no clue why this happens.

Question: raw queries?

To gain some performance in a project we want to do database IO on separate threads asynchronously. The project already heavily includes Twisted so Axiom may be a good fit (we are also thinking about Storm as it has Twisted support somewhat).
Currently, all the the queries are plain sql queries (raw). We would like to move towards an ORM approach, but gradually.
Does axiom have functionalities to run normal sql queries asynchronously? Or any documentation on this? We currently use a sqlite3 database if it matters.

Add Twisted to `setup_requires`

Axiom's "install" command requires twisted to be importable already; this is not necessarily the case if Axiom is being installed as a dependency of something else. Adding it to setup_requires should fix the ordering, but unfortunately this means Twisted will be installed by setuptools, not pip; need to do some investigation to see how this works out.

Axiom schema mismatch exception is misleading

The exception message uses onDiskSchema which is the entire schema from the database, resulting in a very large message with mostly irrelevant information. This should probably be persistedSchema instead, which is the actual value that the code raising the exception compared the inMemorySchema against.

Axiom finalization cache does not work on PyPy

Axiom._fincache, the code in Store that uses it, and the relevant tests rely on objects being deallocated and weakref finalizers being called at very specific times. On PyPy, these events do not occur at quite the same times that the code expects, resulting in errors. I believe it is possible to cope with this by adjusting the way the code uses the cache; while items may still linger in the cache for longer than they would have on CPython, I do not think this should cause any application-level breakage except in application code that is doing extremely unsupported things.

Imported from Launchpad using lp2gh.

Axiom's tox.ini sets PYTHONPATH, which it shouldn't

This potentially hides issues since we will end up using the not-installed package instead of the installed package. idnar believes this is why axiom.test.historic is spuriously succeeding, even though the required extra files are no longer in the MANIFEST.in.

Imported from Launchpad using lp2gh.

Scheduled events aren't automatically executed

This was filed in the old tracker as ticket 3000.

When axiomatic starts up and starts its store service, it fails to do anything which would cause the scheduler service to start. The scheduler service will start running if something causes it to load, but no events are executed until that happens.

Imported from Launchpad using lp2gh.

Incorrect extension of PYTHONPATH

If the PYTHONPATH environment variable is already empty, doing something like:

PYTHONPATH="/path/to/dir:$PYTHONPATH" python [...]

results in a PYTHONPATH of "/path/to/dir:". As the empty segment at the end is interpreted as . (the current directory), this is undesireable. The easiest way to avoid this is to do:

PYTHONPATH="/path/to/dir${PYTHONPATH:+:$PYTHONPATH}" python [...]

This defect is present in axiom.test.test_xatop in Axiom, as well as in epsilon.release in Epsilon.

Imported from Launchpad using lp2gh.

Twisted 15.3.0 MRO breakage 

  File "/home/mithrandi/code/axiom/axiom/scheduler.py", line 230, in <module>
    class _SiteScheduler(object, Service, SchedulerMixin):
  File "/home/mithrandi/deployment/virtualenvs/Axiom/local/lib/python2.7/site-packages/zope/interface/advice.py", line 141, in advise
    newClass = meta(name,bases,cdict)
exceptions.TypeError: Error when calling the metaclass bases
    Cannot create a consistent method resolution
order (MRO) for bases object, Service, SchedulerMixin

This happens because Service inherits from object in Twisted now.

ROWID stability

Currently, Axiom uses the ROWID feature of sqlite (although it calls it "oid" everywhere).

Unfortunately, this column is not explicitly declared anywhere. When this column is not present in the user-declared schema, the VACUUM command will arbitrarily re-write the values in the ROWID column which will irreparably corrupt an Axiom Store.

Worse yet, you can't upgrade these columns with an ALTER TABLE statement, because ALTER TABLE may not specify a PRIMARY KEY or UNIQUE constraint.

However, being unable to VACUUM a database is a serious loss of functionality in SQLite. It basically means that axiom stores will never shrink in size even if most of their data is deleted. VACUUM is the sort of thing you ought to be able to do automatically.

query(orderBy=...).deleteFromStore makes SQLite explode

<class 'sqlite3.OperationalError'>: ORDER BY without LIMIT on DELETE

deleteFromStore should probably just discard the ordering if there is no limit. (It doesn't matter what order you delete them in if you're deleting everything, obviously).

An argument can be made that you just "shouldn't do this", but it turns out to be quite useful to pass query objects around; an API that returns a query with some particular ordering thus can't be used to delete the things, you would have to inefficiently loop over the query and delete each item individually.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.