mrchocha / memphis

Memphis.dev is an intelligent, frictionless message broker. Made to enable developers to build real-time and streaming features fast.

Memphis is an intelligent, frictionless message broker.
Made to enable developers to build real-time and streaming apps fast.

Cloud - Docs - Twitter - YouTube

Memphis.dev is more than a broker. It's a new streaming stack.

It significantly accelerates the development of real-time applications that require a streaming platform with
high throughput, low latency, easy troubleshooting, fast time-to-value,
minimal platform operations, and all the observability you can think of.

🫣 A world without Memphis

When your application requires a message broker or a queue,
Implementing one will require you to -

• Build a dead-letter queue, create observability, and a retry mechanism
• Build a scalable environment
• Create client wrappers
• Tag events to achieve multi-tenancy
• Enforce schemas and handle transformations
• Handle back pressure. Client or queue side
• Configure monitoring and real-time alerts
• Create a cloud-agnostic implementation
• Create config alignment between production to a dev environment
• Spent weeks and months learning the internals through archival documentation, ebooks, and courses
• Onboard your developers
  And the list continues...
  

Or, you can just use Memphis and focus your resources on tasks that matter 😎

✨ Key Features v1.3.0

Roadmap

Production-ready message broker in under 3 minutes
Easy-to-use UI, CLI, and SDKs
Data-level observability
Dead-Letter Queue with automatic message retransmit
Schemaverse - Embedded schema management for produced data (Protobuf/JSON/GraphQL/Avro)
Graph visualization
Storage tiering
SDKs: Node.JS, Go, Python, Typescript, NestJS, REST, .NET, Kotlin
Kubernetes-native
Community driven

Public case studies

• Gastromatic - Synchronizing data using Memphis.dev
• KELA - Real-time cyber threats identification
• Handling millions of discord messages

🚀 Getting Started

Helm for Kubernetes☸

helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && \
helm install my-memphis memphis/memphis --create-namespace --namespace memphis

Docker🐳 Compose

curl -s https://memphisdev.github.io/memphis-docker/docker-compose.yml -o docker-compose.yml && \
docker compose -f docker-compose.yml -p memphis up

Tutorials
Installation videos

High-Level Architecture

Local access

Via Kubernetes

To access Memphis using UI/CLI/SDK from localhost, run the below commands:

  - kubectl port-forward service/memphis 6666:6666 9000:9000 7770:7770 --namespace memphis > /dev/null &

For interacting with the broker via HTTP:

  - kubectl port-forward service/memphis-rest-gateway 4444:4444 --namespace memphis > /dev/null &

Dashboard/CLI: http://localhost:9000
Broker: localhost:6666 (Client Connections)
REST gateway: localhost:4444 (Data + Mgmt)

For Production Environments Please expose the UI, Cluster, and Control-plane via k8s ingress / load balancer / nodeport

Via Docker

Dashboard/CLI: http://localhost:9000
Broker: localhost:6666

SDKs supported features

Feature	Go	Python	JS	.NET	Java	Rust
Connection	✅	✅	✅	✅	✅	✅
Disconnection	✅	✅	✅	✅	✅	✅
Create a station	✅	✅	✅	✅	❌	✅
Destroy a station	✅	✅	✅	✅	❌	✅
Retention	✅	✅	✅	✅	❌	✅
Retention values	✅	✅	✅	✅	❌	✅
Storage types	✅	✅	✅	✅	❌	✅
Create a new schema	✅	✅	✅	✅	❌	❌
Enforce a schema Protobuf	✅	✅	✅	✅	❌	❌
Enforce a schema Json	✅	✅	✅	✅	❌	🚧 (WIP)
Enforce a schema GraphQL	✅	✅	✅	✅	❌	❌
Enforce a schema Avro	✅	✅	✅	✅	❌	❌
Detach a schema	✅	✅	✅	✅	❌	❌
Produce	✅	✅	✅	✅	✅	✅
Add headers	✅	✅	✅	✅	❌	✅
Async produce	✅	✅	✅	✅	❌	✅
Message ID	✅	✅	✅	✅	?	✅
Destroy a producer	✅	✅	✅	✅	Partial	✅
Consume	✅	✅	✅	✅	✅	✅
Context to message handler	✅	✅	✅	✅	❌	Not Applicable
Ack a message	✅	✅	✅	✅	✅	✅
Fetch	✅	✅	✅	✅	✅	❌
Message delay	✅	✅	✅	✅	❌	✅
Get Headers	✅	✅	✅	✅	❌	✅
Get message sequence number	✅	✅	✅	✅	✅	✅
Destroying a Consumer	✅	✅	✅	✅	❌	✅
Check if broker is connected	✅	✅	✅	✅	✅	✅
Consumer prefetch	✅	❌	❌	✅	❌	✅

👉 Use-cases

• Async task management
• Real-time streaming pipelines
• Data ingestion
• Cloud Messaging
  • Services (microservices, service mesh)
  • Event/Data Streaming (observability, analytics, ML/AI)
• Queuing
• N:N communication patterns
• Ingest Grafana Loki logs at scale

Support 🙋‍♂️🤝

Ask a question ❓ about Memphis.dev. or something related to us:

We welcome you to our discord server with your questions, doubts and feedback.

Create a bug 🐞 report

If you see an error message or run into an issue, please create bug report. This effort is valued and it will help all Memphis{dev} users.

Submit a feature 💡 request

If you have an idea, or you think that we're missing a capability that would make development easier and more robust, please Submit feature request.

If an issue❗with similar feature request already exists, don't forget to leave a "+1". If you add some more information such as your thoughts and vision about the feature, your comments will be embraced warmly :)

Contributing

Memphis.dev is an open-source project.
We are committed to a fully transparent development process and appreciate highly any contributions.
Whether you are helping us fix bugs, proposing new features, improving our documentation or spreading the word - we would love to have you as part of the Memphis.dev community.

Please refer to our Contribution Guidelines and Code of Conduct.

Contributors ✨

Thanks goes to these wonderful people ❤:

License 📃

Memphis is open-sourced and operates under the "Memphis Business Source License 1.0" license Built out of Apache 2.0, the main difference between the licenses is: "You may make use of the Licensed Work (i) only as part of your own product or service, provided it is not a message broker or a message queue product or service; and (ii) provided that you do not use, provide, distribute, or make available the Licensed Work as a Service. A “Service” is a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work to third parties as a software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other similar services that compete with Licensor products or services." Please check out License to read the full text.