memphisdev / memphis Goto Github PK

Summary

"Factory" object not making sense and adding a layer of complexity for no good reason.
Should be removed.

Context

The initial thought was to add an extra level of management. After engaging with users, it turned out it's not being used in a way it makes sense for it to stay.

Value

Reduce complexity

Persona(s)

• Data Engineers
• Developers

Describe the bug
A certain station that already exists keep saving the old IP address of a certain producers/consumers which I used their names but from an entirely different host with a different IP address.

To Reproduce
Steps to reproduce the behavior:

1. Create a station
2. Connect a producer with name X
3. Disconnect the producer
4. Connect with the same producer name and same creds to the same station from a different host

Expected behavior
IP should be updated

Summary

Enhanced user management, including permissions per station/features, groups, roles

Context

To enable potential users and organizations with branched teams and different personas that require a higher level of security, compliance, and granularity. In most brokers, ACL (Access control list) mechanism will be found. Memphis will enable the same granularity with ease-of-use

Value

Better security features and compliance

Persona(s)

• DevOps
• Security teams
• Data Engineers

Out of Scope

At first login, the response from the server takes a while.

consumer code

const memphis = require('memphis-dev');

(async function () {
    try {
        await memphis.connect({
            host: 'localhost',
            username: 'root',
            connectionToken: 'memphis'
        });

        const consumer = await memphis.consumer({
            stationName: 'benchmark',
            consumerName: 'yaniv_cons1',
            consumerGroup: ''
        });

        consumer.on('message', (message) => {
            console.log(message.getData().toString());
            message.ack();
        });

        consumer.on('error', (error) => {
            console.log(error);
        });
    } catch (ex) {
        console.log(ex);
        memphis.close();
    }
})();

Try to edit your avatar and then call to refreshToken -> the old avatarId return.

I was looking through the modules and noticed NATS. Isn't NATS by itself claiming to be a better queue to kafka, how does this library add to that?

Summary

The "Getting Started" model for new users is not attracting enough usage and should be redesigned.

Context

Help new users onboard Memphis

Value

Increase engagement and familiarity with Memphis features.

Persona(s)

• Data Engineers
• Developers
• DevOps

Summary

Enable connectivity between Debezium to Memphis to enable CDC use cases.

Context

Multiple streaming pipelines require support in CDC for different databases. Debezium gathers the most popular ones like MySQL, Postgre, and more in a single framework

Value

Faster creation of CDC-based streaming pipelines

Persona(s)

• DevOps
• Data Engineers
• Developers

Asked by

• Yehuda Korotkin

Released on v1.1 and can be found on both the integration center and detailed here as well - https://memphis.dev/blog/part-1-integrating-debezium-server-and-memphis-dev-for-streaming-change-data-capture-cdc-events/

Summary

Create a terraform file to automatically deploy best practiced EKS with Memphis cluster

Context

Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds

Value

One-click installation of Memphis

Persona(s)

• Data Engineers
• DevOps

Summary

Create a terraform file to automatically deploy best practiced DO Kubernetes cluster with Memphis cluster

Context

Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds

Value

One-click installation of Memphis

Persona(s)

• Data Engineers
• DevOps

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
To be able to create stations with names like "demo-1" / "demo.1"

Describe alternatives you've considered
Like rabbitmq routing keys

Hey Dear Contributors,
In the latest steering meeting, users raised a few requests to enable a quick deployment over AWS/GCP/DO,
including Kubernetes cluster with three nodes, Memphis cluster, and ingress for production usage.
The purpose behind it is to make Memphis even easier to deploy. Some devs or users in general are a bit backed down regarding k8s. We would love to lower that entrance barrier.

Referring Material -

https://www.digitalocean.com/community/tutorials/how-to-use-terraform-with-digitalocean
https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs/resources/kubernetes_cluster
https://registry.terraform.io/providers/hashicorp/aws/latest
https://registry.terraform.io/providers/hashicorp/google/latest

Terraform files should be able to -

• Create a K8S cluster over DO / AWS / GCP
• Enable dynamic insertion of API keys / IAM users
• Enable dynamic insertion of cluster name, region, and number of nodes (default 3)
• Ability to deploy nginx ingress as well
• Deploy Memphis when the cluster configuration is finished

If anything is missing or should be added, please comment.

Thank you 💛

From:
08-05-2022 01:46:46 PM - fatal: Memphis - You have to connect with application type user
To:
Memphis - Please use a user of type "Application" and not "Management"

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
At the moment, the DLS feature is enabled by default for every station and cannot be disabled.
We want to add the ability to enable/disable DLS in a certain station.

Motivation

Different schemas often occur when working with multiple data sources / different services / different pipelines that produce data on the same topic, while the consumers on the other side expect specific structure.
Usually, that job is distributed across different teams and business units and requires expensive sync in time and effort.
Upstream changes are often impossible.
For example, AWS Lambda cannot ingest JSON-based payload with a boolean value, but rather should be converted into string before.

Summary

Ability to perform deep, inline transformation of events structure at the broker level while in transit.
Functions like: Concat / trim / regex / sum / and more...
Would be great to get comments about expected functions

Value

Reduce logic from producers and consumers, increase data quality, reduce downtime and crashes, and enable complex streaming pipelines for in-app purposes.

Persona(s)

• Data Engineers
• Developers
• Data Analysts

We need to replace the name of volumes with a meaningful name.
From '89d900b16b4e9cf527b6e6e10dae96ff266ebea390cb208392a335717f191e3c...' To 'Memphis_data' for example.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

When using Stations with the Memory storage type and with a single replica, if we kill the broker and start it again the Station(stream) will perish. although the DB still holds metadata as if the Station exists.

Expected behavior
We should carefully design a solution here,
we can update the DB in this case, or not allow the Memory storage type at all...

Hey,
I have installed Memphis over k8s, the connection token should be unique and based on the generated connection token per installation

The current behavior should only be on Docker installation

Summary

Memphis's core value is to build streaming use cases faster, with a low amount of code.
This feature is a milestone - building a modular framework, easy-to-modify, adjust, and scale connectors framework
that will eliminate the need to create consumers/producers that only collect and push data from/to popular sources.

Context

Based on multiple requests by our users asking to have ready-to-use connectors to pull and push data from and directly to Memphis or out. The number of connectors cannot lean on Memphis team resources and should be built to enable users to quickly implement their type of connector in case they can't find the connector they need in the Hub.

Value

Build fully operational streaming pipelines in a low-code manner

Persona(s)

• Data Engineers
• Developers

Get 500 when I try to create a new station.
details for recovery:
factory name: melvis factory
user: root

Sending 100000 messages to a 2-node memphis cluster receives a timeout after a certain point.
We need to check how to make an automatic throttling and notify the user for it

Summary

Unique, robust, dynamic, and easy-to-use schema store for Protobufs, Avro, and JSON based data formats.
Will be available to manage via the GUI, SDKs, and REST and with dedicated resources embedded in the broker itself without the need to deploy dedicated tools/system and add extra dependencies and management overhead.

Context

A schema defines how a message or event should look and in which format, creating a contract between producer and consumer that the broker will enforce or be aware of.

• We want to redesign how schema management and serialization work compared to other platforms such Schema Registry and AWS GLUE. Memphis will let the user the option to choose if your broker will enforce the schema or will be able to update dynamically with default values.
• Auto schema creator based on message existing schema.
• Ability to import schemas from git repo with CI/CD integrations to update the schema dynamically using DataOps Methodologies.
• Self-service for personas such as product/other devs/BIs/Analysts

Value

1. Enrich Memphis use cases
2. Help users to decouple business logic from consumers and producers
3. Recude code at the producers/consumers level
4. Reduce management overhead compared to other solution
5. Provide self-service for other persona to work with schemas

Persona(s)

• Active users
• Kafka/SQS/Redis/GCP users
• Data Engineers for building complex pipelines

Out of Scope

• Avro
• JSON Schema
  Will be available in Q4

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Describe the bug
See title

To Reproduce

1. Bring a memphis cluster/server up
2. Create a station
3. Send a message without a header using nats cli nats pub stationName.final message
4. Open UI, get into the station view, click on the message
5. the warning message should appear at the top of the page

Expected behavior
the "SDk" should be "SDK"

Screenshots

Summary

Create a terraform file to automatically deploy best practiced GKE with Memphis cluster

Context

Remove deployment frictions of Memphis Broker and have a faster, best-practiced deployment over the popular clouds

Value

One-click installation of Memphis

Persona(s)

• Data Engineers
• DevOps

Can you stop your annoying e-mail spam? I don't care about whatever the project this is.

Hey,
Please add the default username and password (root/memphis) to the print the CP prints when deployed using docker-compose.
Thanks!

Error keeps showing up even after upgrading the consumer SDK and restart it.
Station is stuck with infinite loop

Hey,
At the moment, memphis doesn't allow producers with the same name at the same time. We need to allow this feature.