- Write once, use many times
- Schema-less
- High performance
- Back pressure
- Simple to use and well documented, so anyone can write their own stream processor
- Sliding time window
- Asynchronous via Python 3.5+
async
/await
andasyncio
- Event driven Architecture / Reactor pattern
- Single-threaded core but compatible with threads
- Compatible with pypy, Just-In-Time compiler capable of boosting Python code performace more then 5x times
- Modularized
#!/usr/bin/env python3
import bspump
import bspump.socket
import bspump.common
import bspump.elasticsearch
class MyPipeline(bspump.Pipeline):
def __init__(self, app):
super().__init__(app)
self.build(
bspump.socket.TCPStreamSource(app, self),
bspump.common.JSONParserProcessor(app, self),
bspump.elasticsearch.ElasticSearchSink(app, self, "ESConnection")
)
if __name__ == '__main__':
app = bspump.BSPumpApplication()
svc = app.get_service("bspump.PumpService")
svc.add_connection(bspump.elasticsearch.ElasticSearchConnection(app, "ESConnection"))
svc.add_pipeline(MyPipeline(app))
app.run()
bspump.amqp
AMQP/RabbitMQ connection, source and sinkbspump.elasticsearch
ElasticSearch connection, source and sinkbspump.file
(plain files, JSON, CSV)bspump.http.client
HTTP client source, WebSocket client sinkbspump.http.web
HTTP server source and sink, WebSocket server sourcebspump.influxdb
InfluxDB connection and sinkbspump.kafka
Kafka connection, source and sinkbspump.mysql
MySQL connection, source and sinkbspump.parquet
Apache Parquet file sinkbspump.avro
Apache Avro file source and sinkbspump.socket
TCP source, UDP sourcebspump.mongodb
MongoDB connection and lookupbspump.slack
Slack connection and sinkbspump.trigger
Opportunistic, PubSub and Periodic triggersbspump.crypto
Cryptography- Hashing: SHA224, SHA256, SHA384, SHA512, SHA1, MD5, BLAKE2b, BLAKE2s
- Symmetric Encryption: AES 128, AES 192, AES 256
bspump.analyzer
- Time Window analyzer
- Time Drift analyzer
bspump.lookup
- GeoIP Lookup
Google Sheet with technological compatiblity matrix: https://docs.google.com/spreadsheets/d/1L1DvSuHuhKUyZ3FEFxqEKNpSoamPH2Z1ZaFuHyageoI/edit?usp=sharing
BSPump is an open-source software, available under BSD 3-Clause License.
BSPump is a part of Black Swan project, joint effort of Liberty Aces and TeskaLabs.