tinybirdco / mockingbird Goto Github PK

We should transparently support all Faker types.

A user should be able to use "type": "buildingNumber" and it just works.

Suggest that, when we get a type, we

1. Lookup the type to see if its a Faker type
2. If not, lookup to see if its a custom type

We should also support Faker types with params, consistently with how we support params for custom types today.

E.g. https://fakerjs.dev/api/address.html#countrycode

These should work

{
	"test": {
		"type": "countryCode",
	}
}

{
	"test": {
		"type": "countryCode",
		"params": {
			"alphaCode": "alpha-2"
		}
	}
}

This should fail schema validation because the param does not match what is expected from Faker.

{
	"test": {
		"type": "countryCode",
		"params": {
			"randomParam": 3
		}
	}
}

Additionally, we should go through the current custom types and remove all types that are simply wrappers for Faker types. Any Preset schemas that those those types should be updated to use the standard Faker types.

It's nice to have progress logs, but not in production mode. Could we silence them?

We should disable the ability to make changes to the schema & connection details while generation is active.

When a user clicks 'Start generating' all fields should be disabled.
When the user clicks 'stop generating' all fields are enabled again.

Could we select an existing Tinybird Data Source, get the schema, and guess an appropriate Mockingbird schema? There's no guarantee it's perfect, but could save a user some time to get something working.

When you set your connection detail and click Continue, you can modify the details but there is no way to save them.

E.g

Set limit to 2, click Continue. Change limit to 5.
Start generating, limit will still be 2.

We should enable a user to update the connection settings during a session.

Need a tooltip or some way of explaining to a user what input is expected when selecting 'Custom' in the Tinybird Destination's Host field.

We updated the text inside the preview box to say 'Click Preview...' but the button still says 'Save'.

We should also let the user click Preview as much as they want to keep regenerating the preview.

The 'baggage' selector in the Flight Bookings default schema contains a pickWeighted option, however the installed version of the mock generator doesn't like the given parameters. It needs to be changed to match the current version of the mock generator.

Hi there!

Im trying to generate events to upstash-kafka destination.
Upstash kafka topic was created, and it were receiving events using pure Producer RESR API with success.

But

• If i am trying mockingbird Cloud UI or Local UI - i got failed requests (to https://fit-mallard-XXXXX-eu2-kafka.upstash.io:9092/produce/Orders) in my browser console with no any HTTP status .
• If I am trying CLI: mockingbird-cli upstash-kafka --address=https://fit-mallard-XXXXX-eu2-kafka.upstash.io:9092/ --user=XXXXXX --pass=XXXXXX --topic=Orders --eps=1 --limit=-1 --schema=/Users/vadym/development/mockingbird/schemas/schema.json
  i got Upstash Error: {"message":"request to https://fit-mallard-XXXXX-eu2-kafka.upstash.io:9092/produce/Orders failed, reason: socket hang up","type":"system","errno":"ECONNRESET","code":"ECONNRESET"}.

do i do something wrong or it is some issue?

Name of the service: AWS SNS
Link to the service home page: https://aws.amazon.com/sns/
Link to service's REST API docs: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html

The READMEs are basic/outdated & could be improved

Three READMEs to update:

• Project readme
• Generator readme
• UI readme

I attempted to copy and paste a schema in the "text" option, but it did not allow me to save.

We should look to open up the possible destinations beyond just Tinybird, so that Mockingbird can be used to send data to any HTTP API.

If the event generation was decoupled from the sending logic, we could have 'destination plugins' which simply accept the incoming generated event stream, and can decide how to send the events to the destination. Each destination can also define it's own configuration model for setup.

This way, we could add support for various different flavours of Kafka with different HTTP APIs (Confluent, Upstash, RedPanda) etc.

Each plugin can then be responsible for

1. data formats (json, ndjson, csv)
2. auth styles (Bearer, basic auth, etc)
3. request methods (post, put, etc)

We can take the following steps:

1. De-couple the logic & create a Tinybird destination plugin, so that the feature set is exactly as it is today
2. Re-work the flow for configuration so that a plugin can be selected (lib, cli, ui)
3. Create the first plugin, perhaps for Upstash Kafka

The following schema is useful for simple payment fraud analysis, and used in this blog.
It would be good to have it as a general option in Mockingbird.

{
 "timestamp": {
   "type": "mockingbird.timestampNow"
 },
 "transaction_id": {
   "type": "finance.routingNumber"
 },
 "user_id": {
   "type": "mockingbird.pick",
   "params": [
     {
       "values": ["Mike", "John", "Jane", "Mary", "Bob", "Alice", "Joe", "Sue", "Ryan", "Jill", "Bill","Jen", "Jack", "Jill", "Tom", "Tim", "Tina", "Terry", "Troy", "Tara", "Tanya", "Trevor", "Tucker","Trent", "Trenton", "Mario"]
     }
   ]
 },
 "amount": {
   "type": "finance.amount"
 },
 "ip_address": {
   "type": "internet.ip"
 },
 "browser": {
   "type": "mockingbird.pickWeighted",
   "params": [
     {
       "values": ["Chrome", "Brave", "Firefox", "Safari"],
       "weights": [65, 3, 8, 20]
     }
   ]
 },
 "location": {
   "type": "mockingbird.pickWeighted",
   "params": [
     {
       "values": [ "USA", "Spain", "UK", "Australia", "Mexico", "China", "Japan", "Canada"],
       "weights": [65, 20, 8, 1, 3, 1, 1, 1]
     }
   ]
 },
 "is_declined": {
   "type": "mockingbird.pickWeighted",
   "params": [
     {
       "values": [true, false],
       "weights": [3, 98]
     }
   ]
 },
 "fraud_flag": {
   "type": "mockingbird.pick",
   "params": [
     {
       "values": [false]
     }
   ]
 }
}

This is the tiniest of nitpicks, but the height of the destination buttons is inconsistent between rows 1 and 2, I believe due to the Upstash logo height.

Name of the service: RabbitMQ
Link to the service home page: https://www.rabbitmq.com/, https://www.cloudamqp.com/
Link to service's REST API docs: https://www.cloudamqp.com/docs/api.html, https://www.cloudamqp.com/docs/http.html

Name of the service: Confluent Cloud Kafka
Link to the service home page: https://confluent.io
Link to service's REST API docs: https://docs.confluent.io/cloud/current/api.html#tag/Records-(v3)/operation/produceRecord

tinybirdAPI.ts

There is a 'custom' entry in the endpoint obj that references a env var. If the env var isn't set, the UI doesn't work at all. We should make this an optional setting, so it works OOTB without it

const endpoints = {
  eu_gcp: "https://api.tinybird.co",
  us_gcp: "https://api.us-east.tinybird.co",
  custom: process.env.TB_ENDPOINT,
};

Context

Right now, timestamp type sets a 'recent' date value. We need to have a type that sends exactly the current date, which could be named timestamp_now or date_now, so we can use it in the schema as follows:

"timestamp": {
      "type": "timestamp_now"
}

Design has produced a new style that lines up with the general Tinybird theme, let's update the UI to match. See the internal Figma file.

Let's start there...but I think we can improve the experience beyond this as well.

I was thinking we could turn the 'box' in the middle of the page into some guided drawers, that take the user through Schema Builder -> Destination settings -> Start sending with stats. It can be easy to miss the button in the top corner atm.

When editing a schema, sometimes the predefined schema is reset. Clicking 'save' again clears out any edits, making it incredibly frustrating to modify an existing schema.

Name of the service: Generic HTTP Endpoint (To be able to send to Elasticsearch or any other databases and destinations)
Example from Upstash HTTP HTTP Sink Connector:

It would be great to have a generic HTTP Endpoint destination, for example to send to Elasticsearch over REST HTTP with the possibility to add authorization (username / API key) headers, body, and parameters.

Name of the service: Google Pub/Sub
Link to the service home page: https://cloud.google.com/pubsub
Link to service's REST API docs: https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/publish

King of tiny nitpicks here. "Data source" should be "Data Source"

It's possible that Tinybird can be running on different URLs than the public ones (e.g. in test envs), so we should support a 'custom' endpoint URL.

This needs to be supported across the lib/cli/UI.

For the CLI, we could have a new flag that allows the user to supply a custom URL option.
For the UI, there should be a 'custom' option which then gives the user a free text box to enter a URL.

Name of the service: Ably
Link to the service home page: https://ably.com/
Link to service's REST API docs: https://ably.com/docs/rest/messages

We should update the field labels in the settings modal for the Tinybird destination:

• Data Source
• Auth Token
• Region
• Events Per Second

We should also add some helpers in there too

• Data source: external link arrow icon that links to https://www.tinybird.co/docs/concepts/data-sources.html
• Auth Token: external link arrow icon that links to https://www.tinybird.co/docs/concepts/auth-tokens.html
• Region: a ? icon hoverable tooltip
• Events Per Second: a ? icon hoverable tooltip