warner / foolscap

remote object-messaging for Python+Twisted

Foolscap

Foolscap is an RPC/RMI (Remote Procedure Call / Remote Method Invocation) protocol for use with Twisted, derived/inspired by Twisted's built-in "Perspective Broker" package.

If you have control of both ends of the wire, and are thus not constrained to use some other protocol like HTTP/XMLRPC/CORBA/etc, you might consider using Foolscap.

Fundamentally, Foolscap allows you to make a python object in one process available to code in other processes, which means you can invoke its methods remotely. This includes a data serialization layer to convey the object graphs for the arguments and the eventual response, and an object reference system to keep track of which objects you are connecting to. It uses a capability-based security model, such that once you create a non-public object, it is only accessible to clients to whom you've given the (unguessable) FURL. You can of course publish world-visible objects that have well-known FURLs.

Full documentation and examples are in the doc/ directory.

DEPENDENCIES

• Python 3.8 or higher
• Twisted 16.0.0 or later
• PyOpenSSL (tested against 16.0.0)

INSTALLATION

To install foolscap into your system's normal python library directory, just run the following (you will probably have to do this as root):

pip install .

You can also just add the foolscap source tree to your PYTHONPATH, since there are no compile steps or .so/.dll files involved.

COMPATIBILITY

Foolscap's wire protocol is unlikely to change in the near future.

Foolscap has a built-in version-negotiation mechanism that allows the two processes to determine how to best communicate with each other. The two ends will agree upon the highest mutually-supported version for all their traffic. If they do not have any versions in common, the connection will fail with a NegotiationError.

Please check the NEWS file for announcements of compatibility-breaking changes in any given release.

As of Foolscap-0.14.0, this library is mostly compatible with Python 3 (specifically 3.5 or higher), and is tested against 3.5, 3.6, 3.7, and 3.8. It will retain compatibility with Python 2.7 for a little while longer, to ease the transition, but since Python 2 was marked End-Of-Life in January 2020, this compatibility will not be maintained forever.

HISTORY

Foolscap is a rewrite of the Perspective Broker protocol provided by Twisted (in the twisted.pb package), with the goal of improving serialization flexibility and connection security. It also adds features to assist application development, such as distributed/incident-triggered logging, Service management, persistent object naming, and debugging tools.

For a brief while, it was intended to replace Perspective Broker, so it had a name of "newpb" or "pb2". However we no longer expect Foolscap to ever be added to the Twisted source tree.

A "foolscap" is a size of paper, probably measuring 17 by 13.5 inches. A twisted foolscap of paper makes a good fool's cap. Also, "cap" implies capabilities, and Foolscap is a protocol to implement a distributed object-capabilities model in python.

AUTHOR

Brian Warner is responsible for this thing. Please discuss it on the twisted-python mailing list.

The Foolscap home page is a Trac instance at http://foolscap.lothar.com/trac. It contains pointers to the latest release, bug reports, patches, documentation, and other resources.

Foolscap is distributed under the same license as Twisted itself, namely the MIT license. Details are in the LICENSE file.

foolscap's Issues

txsocksx dependency prevents use with Python 3

It doesn't look like txsocksx is going to be updated for Python 3 (becuase Parsley isn't being updated for Python 3), so having a dependency on that for SOCKS support will preclude the use of Python 3.

txtorcon has already dropped this dependency and so this does not affect Tor support for foolscap.

#ERASE#

#ERASE#

Are there any foolscap performance data?

Are there any foolscap performance data? We are evaluating python RPC, foolscap,grpc etc. It seemed that Foolscap is suitable for our application.

The foolscap website and wiki are down :(

Can you put it up again please?

Relicense some code for Tahoe-LAFS?

Hi,

I want to move some code from https://github.com/warner/foolscap/blob/master/src/foolscap/crypto.py into Tahoe-LAFS. Since @warner you appear to be the sole author, are you willing for it to be relicensed under the Tahoe-LAFS license?

Thanks!

Would you be OK adding additional maintainers?

E.g. me, or someone else you trust—a few small fixes in Foolscap would allow for easier Python 3 port for Tahoe-LAFS.

directly-instantiated types are unsupported in mypy-zope

In Shoobx/mypy-zope#21, I learned that mypy-zope doesn't support direct invocation of an InterfaceClass such as seen here in foolscap. With Shoobx/mypy-zope#24, however, it will support the more typical metaclass syntax. Perhaps the best approach here may be to use the appropriate metaclass instantiation, something like

class RemoteInterface(metaclass=RemoteInterfaceClass): ...

(and an equivalent for Python 2).

port to python 3

Python 2's End-Of-Life date is Jan 1, 2020. Tahoe-LAFS currently depends upon Foolscap, as does git-foolscap and flancer and a handful of personal tools. Tahoe's nominal plan is to switch to a new HTTP-based transport layer and remove Foolscap entirely, but that will introduce backwards-compatibility problems (basically each grid will have a flag day where they must update everybody to the new py3-without-foolscap version of Tahoe at the same time).

I'm not making any scheduling promises, but I'm working on porting Foolscap to python3: watch the py3 branch to follow along. I'm aiming to make it work under both py2 and py3, like I did with magic-wormhole.

tarball unpacks to a strange location

The 23.11.0 tarball unpacks to foolscap-foolscap-23.11.0 instead of the expected foolscap-23.11.0.

Referenceable convention not recognized by mypy

In this comment, I isolated an issue where mypy static analysis checks are failing on a foolscap Referenceable RemoteInterface. Given this minimal implementation:

# remote.py
from foolscap.api import RemoteInterface, Referenceable
from zope.interface import implementer


class RIBucketWriter(RemoteInterface):
    """ Objects of this kind live on the server side. """
    def write(offset=None, data=None):
        return None


@implementer(RIBucketWriter)
class BucketWriter(Referenceable):
    def remote_write(self, offset, data):
        pass

and this mypy configuration:

[mypy]
ignore_missing_imports = True
plugins=mypy_zope:plugin

Then, creating an environment with unmerged branches of mypy-zope and foolscap:

test $ python -m venv env
test $ env/bin/pip install -q mypy-zope@git+https://github.com/jaraco/mypy-zope@bugfix/21-InterfaceClass-subclass foolscap@git+https://github.com/jaraco/foolscap@bugfix/75-use-metaclass

Attempting to run mypy on that simple BucketWriter produces this error:

test $ env/bin/mypy remote.py
remote.py:18: error: 'BucketWriter' is missing following 'RIBucketWriter' interface members: write.
Found 1 error in 1 file (checked 1 source file)

It seems mypy is unaware of the contract that a Referenceable adds providing the required interface but with method names replaced with remote_ prefixed.

Presumably the proper fix would be to add mypy support (plugin) for these Referenceable objects.

There are some I2P leftovers

docs/connection-handlers.rst discusses I2P as if it is still supported. It was removed as part of the Python 3 porting effort because txi2p is Python 2-only.

tor integration not working with recent txtorcon or Tor itself

In the last month or three, I noticed tor-based connections failing with a NoConnectionHintsError (which means all of the supplied hints were unusable). I traced it down to the config.SocksPort returned by txtorcon returning a nested list [["9050", "unix:/var/run/tor/socks WorldWritable"]], whereas previously it was giving me a single list like ["9050", "unix:/var/run/tor/socks WorldWritable"].

I wasn't able to figure out why the format changed, or whether this new behavior was coming from a new version of txtorcon vs a new version of Tor, but the easiest fix seemed to be to tolerate nested lists.

test failures OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_use_certificate', 'ee key too small')]

In debian unstable with openssl 1.1.1 and CipherString = DEFAULT@SECLEVEL=2 in /etc/ssl/openssl.cnf some tests fail with following error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 896, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 316, in debug_fireTimer
    call(*args)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 526, in sendPlaintextServerAndStartENCRYPTED
    self.startENCRYPTED()
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 555, in startENCRYPTED
    self.startTLS(self.tub.myCertificate)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 1091, in startTLS
    self.transport.startTLS(ctxFactory)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 179, in startTLS
    startTLS(self, ctx, normal, FileDescriptor)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 139, in startTLS
    tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 773, in __init__
    contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 651, in __init__
    oldStyleContextFactory.getContext()
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/crypto.py", line 46, in getContext
    ctx = CertificateOptions.getContext(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1650, in getContext
    self._context = self._makeContext()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1660, in _makeContext
    ctx.use_certificate(self.certificate)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 949, in use_certificate
    _raise_current_error()
  File "/usr/lib/python2.7/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_use_certificate', 'ee key too small')]

Setting the CipherString SECLEVEL to one fixes the tests. Are there some certificates in the testsuite that might need regenerating with larger keys or newer algorithms?

These tests all fail:

foolscap.test.test_negotiate.Future.testFuture1
foolscap.test.test_negotiate.Future.testFuture2
foolscap.test.test_negotiate.Future.testFuture3
foolscap.test.test_negotiate.Future.testFuture4
foolscap.test.test_negotiate.Parallel.test2
foolscap.test.test_negotiate.Parallel.test3
foolscap.test.test_negotiate.Parallel.test4
foolscap.test.test_negotiate.Parallel.test5
foolscap.test.test_negotiate.Replacement.testAncientClient
foolscap.test.test_negotiate.Replacement.testAncientClientWorkaround
foolscap.test.test_negotiate.Replacement.testBouncedClient
foolscap.test.test_negotiate.Replacement.testBouncedClient_Reverse
foolscap.test.test_negotiate.Replacement.testConnectionHintRace
foolscap.test.test_negotiate.Replacement.testLostDecisionMessage_NewServer
foolscap.test.test_negotiate.Replacement.testNATEntryDropped
foolscap.test.test_negotiate.Replacement.testTwoLostDecisionMessages
foolscap.test.test_negotiate.Replacement.testWeirdSeqnum
foolscap.test.test_negotiate.SharedConnections.test1
foolscap.test.test_negotiate.ThreeInParallel.test1

Python 3.13 support

Python 3.13 will be coming out this fall, would be good to support it.

Aug 15: Twisted now has working 3.13 support in trunk, still unreleased.

Logging fails with dictionary keys that are bytes

On Python 2, JSON would happily encode byte strings, so long as they were ASCII at least. In Python, the json module knows nothing about bytes.

In Tahoe-LAFS, there is now custom code to deal with this, e.g. https://github.com/tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/util/jsonbytes.py#L31 and later parts of that module.

Foolscap is still lacking this infrastructure. As such, when a log message comes in where keys are bytes, it fails, e.g.:

2021-05-10T10:40:57-0400 [stdout#info] GATHERER: unable to serialize {'facility': 'tahoe.storage_broker', 'format': '%(name)s provided version info %(version)s', 'incarnation': ('86433dff70db2783', None), 'level': 10, 'name': b'q26hhekw', 'num': 626, 'parent': 617, 'time': 1620657650.2063904, 'umid': 'SWmJYg', 'version': {b'application-version': b'tahoe-lafs/1.14.0.post3323', b'http://allmydata.org/tahoe/protocols/storage/v1': {b'available-space': 7235896832, b'delete-mutable-shares-with-zero-length-writev': True, b'fills-holes-with-zero-bytes': True, b'maximum-immutable-share-size': 7235896832, b'maximum-mutable-share-size': 69105000000000000, b'prevents-read-past-end-of-share-data': True, b'tolerates-immutable-read-overrun': True}}}: keys must be a string}

Switch to GitHub Actions for CI

Tox changed

Tox changed the way it accepts certain arguments. Either pin to older or change locally.

pypi lacks sources

The pypi page has a wheel but no sources.

InterfaceClass passes __module__ as __doc__

In this code, the RemoteInterfaceClass.__init__ is calling the parent class, but passes __module__ positionally. Looking at the parent implementation, that position is suited for __doc__. I suspect this mismatch is a bug.

New release

It would be good to have version that supports 3.12 up on PyPI.

saving log events between py2/py3

As mentioned in #48 (comment) , when a py2-based client emits a log event, the receiver (flogtool tail, log-gatherer, incident-gatherer) gets an event dictionary that uses bytes for both the keys and the values. If the receiver is running py3, the json.dumps() will fail, as it is more picky about the key types than the py2 json module was, and insists that the keys are text (str under py3).

The json module has an override (cls= and implement JSONEncoder.default) for handling non-serializable objects, but this doesn't appear to enable the serialization of bytestring keys. The hook isn't implemented for dictionaries at all (nor any other type that it already knows how to serialize).

So to fix this, I think we'd need a recursive rewriter that takes the dictionary, walks through all collections inside it (dicts, but also lists), and returns a new dict with text keys.

For the sake of rendering, it might also be nice to replace bytes values with text equivalents, as most of the values in log events are boring ASCII strings too.

The wrinkle is that application code can provide additional arguments (yay structured logging), and their values are not necessarily boring ASCII. They could contain nested dictionaries, with arbitrary keys. It's probably fair to insist that log events be serializable, even though part of the intended benefit of structured logging was to let the application author record whatever data would be useful in future debugging, without needing to think about how it should be rendered into text.

(I think we were originally using our own Banana serialization for log events, which was more flexible, but more complicated, and managed to introduce dependencies like the log-viewer had to be able to import the log-emitter's classes, eww)

So if a log event arrives over the wire with bytes keys, we should be ok rewriting them to be strings, and just deny things like hash tables with binary keys. If the values are bytes too, we convert them too, perhaps lossily.

cannot make remote call with kwarg named "f"

oops

Support for Python 3.12

At minimum need to upgrade versioneer, but there may be other issues.

AttributeError: module 'inspect' has no attribute 'getargspec'

Foolscap fails its tests with Python 3.11:

  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.11/build/foolscap/remoteinterface.py", line 191, in initFromMethod
    names, _, _, typeList = inspect.getargspec(method)
                            ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'

It seems inspect.getargspec has been deprecated for some time and finally removed in Python 3.11.

test failure with twisted 18.7.0

In debian unstable amd64 with twisted 18.7.0 foolscap currently fails the
foolscap.test.test_negotiate.Crossfire.test4 test

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/build/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/test/test_negotiate.py", line 454, in test4
    rref1 = yield self.connect2()
  File "/build/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/test/test_negotiate.py", line 382, in connect2
    rref1 = yield d1
foolscap.ipb.DeadReferenceError: Connection was lost (to tubid=htje) (during method=RIBroker:getReferenceByName)

foolscap.test.test_negotiate.Crossfire.test4
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/build/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/banana.py", line 639, in dataReceived
    self.handleData(chunk)
  File "/build/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/banana.py", line 739, in handleData
    "but got %r" % s)
foolscap.tokens.BananaError: BananaError(in <RootUnslicer>): ("token prefix is limited to 64 bytes: but got 'banana-decision-version: 1\\r\\nerror: No valid banana-negotiation sequence seen\\r\\n'",)

foolscap.test.test_negotiate.Crossfire.test4
-------------------------------------------------------------------------------

foolscap.test.test_info.Connect.testListener also fails but that is just different output and easily fixed:
#41

--- a/src/foolscap/test/test_info.py
+++ b/src/foolscap/test/test_info.py
@@ -146,9 +146,13 @@ class Connect(unittest.TestCase):
         self.assertEqual(ci.connectorStatuses, {})
         (listener, status) = ci.listenerStatus
         self.assertEqual(status, "successful")
-        self.assertEqual(listener,
-                         "Listener on IPv4Address(TCP, '127.0.0.1', %d)"
-                         % self._portnum)
+
+        self.assertTrue(
+            (listener == ("Listener on IPv4Address(TCP, '127.0.0.1', %d)"
+                          % self._portnum)) or
+            (listener == ("Listener on IPv4Address(type='TCP', host='127.0.0.1',"
+                          " port=%d)"
+                          % self._portnum)))
 
     @defer.inlineCallbacks
     def testLoopback(self):

Restore i2p support

txi2p-tahoe package is now available with Python 3 support, so Foolscap can revive its i2p support.

[feature] Provide type stubs

Downstream projects, including Tahoe-LAFS, are adopting mypy for static type checking and currently must suppress failing typechecks that on foolscap types like IRemoteReference. If foolscap were to provide these stubs or type hints and declare itself as a typed library, that would enhance the ability of consumers to provide type checks on foolscap-derived objects.

Tub.listenOn() accepts strs, not unicodes

Tub.listenOn() accepts type str for the descriptor of how to listen. As we all know, it is an offense against decency and rightness to put human-readable data into strs. It is akin to kicking a basket of kittens. The right thing to do, of course, is to put that human-readable data into unicodes, which is like petting the kittens and giving them a saucer of warm milk.

Automated PyPI releases

Specifically, consider setting up https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/ and then anyone who can tag a release can trigger an upload to PyPI.

callRemote() requires a native string, making Python 3 porting harder

Tahoe-LAFS is being ported to run on both Python 2 and 3 at same time; eventually the former will be dropped, but for now it's both.

We're adding from __future__ import unicode, but that breaks things like remoteref.callRemote("mymethod", arg1) because on Python 2, "mymethod" is now Unicode, but on Python 2 Foolscap expects it to be a native string, i.e. bytes.

It would be good to accept unicode strings on Python 2 as well; I will submit PR that does that.

Release 21.1

Make a new release!

This will include mypy support (#71, #75) and can fix the packaging issue described in #70 (related to which it would be nice to get the fix for #80 in as well)

Wheel release of 20.4.0 on pypi contains old i2p code

i2p should have gone in foolscap 20.4.0, but at least foolscap/connections/i2p.py is present in the .whl file. The corresponding .zip file does not contain that.

Quick test:

$ wget https://files.pythonhosted.org/packages/82/f8/be775e689bd579b102a9b7aa731fac8726e27f97a2659481098488e3eac3/foolscap-20.4.0-py2.py3-none-any.whl
$ unzip foolscap-20.4.0-py2.py3-none-any.whl
$ find . -name "i2p.py"
./foolscap/connections/i2p.py

(This is related to tahoe-lafs/tahoe-lafs#700)

Set up Windows CI

See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1397

None of Appveyor, Microsoft Azure, and GitHub Actions existed when that ticket was originally filed and so getting Windows CI was a lot harder. Now there are several options. Foolscap should have Windows CI of its own.