Coder Social home page Coder Social logo

crossbario / autobahn-python Goto Github PK

View Code? Open in Web Editor NEW
2.5K 100.0 765.0 19.06 MB

WebSocket and WAMP in Python for Twisted and asyncio

Home Page: https://crossbar.io/autobahn

License: MIT License

Python 95.76% Makefile 1.25% C 0.68% Shell 0.15% Jinja 2.16%
python websocket real-time wamp rpc pubsub autobahn

autobahn-python's Introduction

Autobahn|Python

WebSocket & WAMP for Python on Twisted and asyncio.

Version CI Test Status CI Deploy Status CI Docker Status CI EXE Status Docs Docker Images EXE Download

Introduction

Autobahn|Python is a subproject of Autobahn and provides open-source implementations of

for Python 3.7+ and running on Twisted and asyncio.

You can use Autobahn|Python to create clients and servers in Python speaking just plain WebSocket or WAMP.

WebSocket allows bidirectional real-time messaging on the Web and beyond, while WAMP adds real-time application communication on top of WebSocket.

WAMP provides asynchronous Remote Procedure Calls and Publish & Subscribe for applications in one protocol running over WebSocket. WAMP is a routed protocol, so you need a WAMP Router to connect your Autobahn|Python based clients. We provide Crossbar.io, but there are other options as well.

Note

Autobahn|Python up to version v19.11.2 supported Python 2 and 3.4+, and up to version v20.7.1 supported Python 3.5+, and up to version v21.2.1 supported Python 3.6+.

Features


Show me some code

To give you a first impression, here are two examples. We have lot more in the repo.

WebSocket Echo Server

Here is a simple WebSocket Echo Server that will echo back any WebSocket message received:

from autobahn.twisted.websocket import WebSocketServerProtocol
# or: from autobahn.asyncio.websocket import WebSocketServerProtocol

class MyServerProtocol(WebSocketServerProtocol):

    def onConnect(self, request):
        print("Client connecting: {}".format(request.peer))

    def onOpen(self):
        print("WebSocket connection open.")

    def onMessage(self, payload, isBinary):
        if isBinary:
            print("Binary message received: {} bytes".format(len(payload)))
        else:
            print("Text message received: {}".format(payload.decode('utf8')))

        # echo back message verbatim
        self.sendMessage(payload, isBinary)

    def onClose(self, wasClean, code, reason):
        print("WebSocket connection closed: {}".format(reason))

To actually run above server protocol, you need some lines of boilerplate.

WAMP Application Component

Here is a WAMP Application Component that performs all four types of actions that WAMP provides:

  1. subscribe to a topic
  2. publish an event
  3. register a procedure
  4. call a procedure
from autobahn.twisted.wamp import ApplicationSession
# or: from autobahn.asyncio.wamp import ApplicationSession

class MyComponent(ApplicationSession):

    @inlineCallbacks
    def onJoin(self, details):

        # 1. subscribe to a topic so we receive events
        def onevent(msg):
            print("Got event: {}".format(msg))

        yield self.subscribe(onevent, 'com.myapp.hello')

        # 2. publish an event to a topic
        self.publish('com.myapp.hello', 'Hello, world!')

        # 3. register a procedure for remote calling
        def add2(x, y):
            return x + y

        self.register(add2, 'com.myapp.add2')

        # 4. call a remote procedure
        res = yield self.call('com.myapp.add2', 2, 3)
        print("Got result: {}".format(res))

Above code will work on Twisted and asyncio by changing a single line (the base class of MyComponent). To actually run above application component, you need some lines of boilerplate and a WAMP Router.

Extensions

Networking framework

Autobahn runs on both Twisted and asyncio. To select the respective netoworking framework, install flavor:

  • asyncio: Install asyncio (when on Python 2, otherwise it's included in the standard library already) and asyncio support in Autobahn
  • twisted: Install Twisted and Twisted support in Autobahn

WebSocket acceleration and compression

  • accelerate: Install WebSocket acceleration - Only use on CPython - not on PyPy (which is faster natively)
  • compress: Install (non-standard) WebSocket compressors bzip2 and snappy (standard deflate based WebSocket compression is already included in the base install)

Encryption and WAMP authentication

Autobahn supports running over TLS (for WebSocket and all WAMP transports) as well as WAMP-cryposign authentication.

To install use this flavor:

  • encryption: Installs TLS and WAMP-cryptosign dependencies

Autobahn also supports WAMP-SCRAM authentication. To install:

  • scram: Installs WAMP-SCRAM dependencies

XBR

Autobahn includes support for XBR. To install use this flavor:

  • xbr:

To install:

pip install autobahn[xbr]

or (Twisted, with more bells an whistles)

pip install autobahn[twisted,encryption,serialization,xbr]

or (asyncio, with more bells an whistles)

pip install autobahn[asyncio,encryption,serialization,xbr]

Native vector extensions (NVX)

> This is NOT yet complete - ALPHA!

Autobahn contains NVX, a network accelerator library that provides SIMD accelerated native vector code for WebSocket (XOR masking) and UTF-8 validation.


WAMP Serializers

  • serialization: To install additional WAMP serializers: CBOR, MessagePack, UBJSON and Flatbuffers

Above is for advanced uses. In general we recommend to use CBOR where you can, and JSON (from the standard library) otherwise.


To install Autobahn with all available serializers:

pip install autobahn[serializers]

or (development install)

pip install -e .[serializers]

Further, to speed up JSON on CPython using ujson, set the environment variable:

AUTOBAHN_USE_UJSON=1

Warning

Using ujson (on both CPython and PyPy) will break the ability of Autobahn to transport and translate binary application payloads in WAMP transparently. This ability depends on features of the regular JSON standard library module not available on ujson.

autobahn-python's People

Contributors

abrandl avatar agronholm avatar alan-ls avatar codelectron avatar dimbleby avatar flyser avatar hawkowl avatar izderadicka avatar jameshilliard avatar jsoref avatar klemmster avatar luhn avatar meejah avatar mriehl avatar oberstet avatar om26er avatar oopman avatar potens1 avatar remyroy avatar rndwb avatar rogererens avatar rusydi avatar sametmax avatar skully17 avatar sprocket-9 avatar terrycojones avatar timgates42 avatar vstinner avatar william-p avatar zaphoyd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

autobahn-python's Issues

limits: send 2**63 frame

sending a 2^63 frame using streaming producer .. case must use streaming API.

check if and when the peer implementation bails out.

Reports Database

Persistently store all run cases / reports into SQLite.

Currently, everything is stored only in run-time Python objects, and when the fuzzer ends, that state is lossed. Users have to run everything they want to test in one session, and there is no history of reports.

utf-8 fail fast

UTF-8 fail fast => differentiate 4 behaviors:

  1. only when complete message was received
  2. only when complete frame was received
  3. only when complete octets for encoded integer was received
  4. as soon as invalid octet is received

Currently, there are tests to distinguish between 1., 2. and 4.

So for completeness, there need to be tests to differentiate between 3. and 4.

overlong payload length

peer must fail on overlong payload length encodings:

send frames having non-minimal payload length encoding, check that peer fails

make invalid close frame handling consistent

receiving a close frame with payload length 1 will always drop the TCP, while payload length 126 will respond with 1002 when failByDrop = False

the response to a protocol violation during the closing handshake, here when peer iniated close and we receive a close frame with invalid payload length should be

  1. either 1002 (failByDrop = False) or
  2. drop tcp (failByDrop = True)

make sure this is how AB behaves.

same with

  • invalid utf-8 as close reason
  • invalid (reserved) close code

Thus:

invalid payload length => 1002 or drop
reserved close code => 1002 or drop
invalid close reason => 1002 or 1007 (?) or drop

For the last one, decide which of 1002 or 1007 to do:

Argument for 1007: "one sends the most specific reason, the other sends the least"
Argument for 1002: "somehow more consistent with the others"

Go with 1007!

But allow both 1002 and 1007 for the test cases.

sendClose API

sendClose => 3 variants are needed:

  1. user level
  2. WS-internal level
  3. fuzzing/raw level

"1." restricts close code to 1000 and 3000-4999 like the browser client api

"2." restricts close code to [CLOSE_STATUS_CODE_NORMAL,
CLOSE_STATUS_CODE_GOING_AWAY,
CLOSE_STATUS_CODE_PROTOCOL_ERROR,
CLOSE_STATUS_CODE_UNSUPPORTED_DATA,
CLOSE_STATUS_CODE_INVALID_PAYLOAD,
CLOSE_STATUS_CODE_POLICY_VIOLATION,
CLOSE_STATUS_CODE_MESSAGE_TOO_BIG,
CLOSE_STATUS_CODE_MANDATORY_EXTENSION]

"3." does not restrict close code at all

HTTP authentication

support HTTP authentication:

  • basic
  • digest
  • cram-md5, cram-sha1, cram-sha256

test suite exception generating reports with timeouts

On cases where the test suite reaches it's timeout on client tests (ex: test 1.1.6) it writes a KLE type message to the wire log. The report generator can't handle this and throws an exception.

File "/Library/Python/2.7/site-packages/autobahn-0.4.3-py2.7.egg/autobahn/fuzzing.py", line 1002, in createAgentCaseReport
        raise Exception("logic error (unrecognized wire log row type %s - row %s)" % (t[0], str(t)))
    exceptions.Exception: logic error (unrecognized wire log row type K - row KLE)

I believe the offending line is here:

https://github.com/oberstet/Autobahn/blob/master/lib/python/autobahn/fuzzing.py#L203

self.wirelog.append(("KLE"))

should be

self.wirelog.append(("KLE", )) ?

forward onCloseFrame to Case

forward the onCloseFrame we introduced in websocket.py via fuzzing.py to the case - at least to have the option to override in the case

extended close codes

Extended Close Codes: Reason + Reconnect (ms)

  • internal service error
  • service restart
  • service overload

reserved close code behavior

assumption: using reserved close code values is a protocol violation.

reserved close codes = 0-999, 1004, 1005, 1006, 1011-2999, >= 5000

the response to a protocol violation during the closing handshake, here when peer iniated close and we receive a close frame with a reserved close code is

  1. either 1002 (failByDrop = False) or
  2. drop tcp (failByDrop = True)

make sure this is how AB behaves

extended testee protocol

Probably, we need to have a way to instruct the tested peer to iniate a closing handshake to test certain things (like timeouts).

Currently, a tested implementation only needs to echo back any message (text or binary) it has received.

We slightly extend that: when a text message is received with first char = '#', then the rest is a command, like i.e. "#close".

A tested implementation that only implements the echo would echo back the command, and the test case sending the command could recognize that the peer does not implement the extended testing commands.

standard testee protocol: peer must echo anything (like today)

extended testee protocol: as above, unless msg = text, and msg[0] == "#"

the rest msg[1:] should be the command: comma separated key:value pairs

where key, value = us-ascii7 without "," or ":"

when a testee does not understand the extended protocol, it'll just echo .. so the fuzzer/case can then recognize that the peer does not speak the "extended test protocol".

open: msg[0] == "#"

=> test protocol command ... either from peer-to-fuzzer or fuzzer-to-peer

is there any conceivable scenario where the tested peer wants to "command" the fuzzer?

note that we want to have "direct mode" also .. so above would only be : does the tested peer want to "command" the fuzzer when running in "case mode"?

Allow test suite usage without installing the package

I'm using Auobahn's test suite to test Webbit's WebSocket implementation. In order to easily update the test suite when it changes we're thinking about including it as a git submodule.

It would be nice if we could run it without having to install the package, simply by setting PYTHONPATH. This is currently not possible since lib/python/autobahn/fuzzing.py uses pkg_info to get the autobahn_version.

Would you be willing to add a fallback solution for this? For example falling back to "UNKNOWN" or something if the package is not installed?

Cheers,
Aslak

close - multiple close frames

test sending 2 or more close frames: peer must only respond to first with close reply and ignore (fail?) the others

close behavior descr in reports

in the detail report table for close behavior, add a 3rd column with description of the rows taken from the comments in the code

close - protocol violation after close hs start

make peer initiate close ("extended test protocol"), but respond with something violating the protocol (i.e. a fragmented close reply, a close frame with invalid code/reason).

see how the peer acts when confronted with this "double error situation"

WS settings page

generate settings page for report (with WS protocol option settings)

direct command mode

In "case mode", a test case will completely be driven by the case implementation class running in the fuzzing client or server.

In "direct mode", the testee will command the fuzzing client or server to "fuzz back" by specifying what to do .. including things that break the WS spec.

close - pings/msgs after close

make the peer initiate close, but then ignore that and continue by sending messages/pings.

does the peer reply to those?

does it time out on his close iniation and fail the connection?

don't send / ignore receive after close

The spec says:

"""
After sending a control frame indicating the connection should be
closed, a peer does not send any further data; after receiving a
control frame indicating the connection should be closed, a peer
discards any further data received.
"""

Make sure AB behaves like this under all conditions.

split 6.4.2

split out 6.4.3 to a new test section: "Implementation Limits" => see how behaves

Testsuite Web UI

The fuzzing_server already runs an embedded web server (Twisted Site).

This could be extended to allow configuration/access to "Reports Database" (see issue #46), and also to interactively start fuzzing_clients.

frame/streaming API - FIN in beginMessageFrame

allow FIN = true on beginMessageFrame instead of endMessage

currently, with the frame-based/streaming API, the final frame is only marked in endMessage which means the last frame will have payload = 0

JSON roundtrips

JSON roundtrip tests ... need to see if that is possible in that generality

limit parallel connection attempts

The spec says:

"There MUST be no more than one connection in a CONNECTING state. If multiple connections to the same IP address are attempted simultaneously, the client MUST serialize them"

openingHandshakeTimeout

a client should timeout upon not receiving a handshake reply
server should timeout after tcp connect, but no handshake request

probably fail both client and server simply on "handshake not finished within X secs"

variant of 6.4.2

add new case variant of 6.4.2 where the middle frame is large (64k)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.