Comments (17)
Batchers are about grouping data before "publishing" it to a separate step. So for example, if you want write to RabbitMQ one by one and write to the database ten by ten, you define two batchers.
So 10 different destinationis is totally possible today. 10 processing steps aren't right now but this is tracked in #39.
from broadway.
Right, you can't do accumulator in Broadway, so if that is a must, Flow is a better option! But we can allow hashing, which will at least guarantee the same processors handle the same data. This can be useful at least for ordering. I have opened #62 for this case. Thanks!
from broadway.
Hi @am-kantox. Thanks for your proposal!
We have already discussed some ideas to make Broadway more flexible/pluggable in the near future. However, we decided, for now, to get more feedback like this from the community before we make a decision in that direction.
We certainly going to get back to this issue in the next few days/weeks, so stay tuned and thanks again for your feedback!
from broadway.
Hi @am-kantox! Thanks for feedback.
I have some questions about your current pipeline.
When you say you have RabbitMQ -> Flow -> RabbitMQ -> Flow
, do you mean that:
-
you get the data from RabbitMQ, then you process it with Flow, then you send it to RabbitMQ, and then there is another Flow that gets the data from RabbitMQ again?
-
or you get the data from RabbityMQ, then you process it with Flow, then you send it to RabbitMQ, and continue processing it with Flow (without getting it from RabbitMQ again)?
That will help us steer the discussion in the correct direction.
from broadway.
from broadway.
The first one. We try to decouple all Flows as much as possible, hence no ties.
@am-kantox so in this case, you should be able to create multiple broadway pipelines. Correct? Why would publishers be necessary? Or is just that you don't need batching?
from broadway.
from broadway.
That does mean that Broadway might take care about number of publishers to Rabbit, or about number of requests per a sec to some other service, assuming we have peak times and quiet times.
from broadway.
There are two ways we can implement this functionality today with GenStage:
-
You have custom consumers that set the demand to manual to have granular control over the demand
-
You simply block in the consumer until the thing you are calling can accept the request
Almost everyone is doing 2, because it is by far the simplest approach, and that's what Broadway supports. It is also not a major issue to block a consumer.
Have you implemented 1 by any chance? If so, I would love to take a look at it, otherwise, I would postpone this until we have use cases in place.
from broadway.
from broadway.
Still, the ability to deliberately select a publish method according to the endpoind is valuable IMHO.
You can achieve this today by setting up multiple batchers. :)
from broadway.
from broadway.
I am not a mind reader :)
Glad to hear this title means exactly this.
from broadway.
@am-kantox the batchers part though (10 destinations) exists already today and is in the docs. Was that not made clear in the docs? Is there something we could improve?
from broadway.
from broadway.
If you are using Broadway, I don't see why you would use Flow. We already do all of the partitioning, parallelism, etc for you. Given you said you are doing RabbitMQ -> Flow -> RabbitMQ -> Flow
this will be two broadway pipelines: one that gets from RabbitMQ, processes it and sends to another RabbitMQ channel. And another pipeline that gets from RabbitMQ as well and consumes it.
Why would you need both Broadway and Flow?
from broadway.
Flow
is a more general and powerful abstraction thanBroadway
that focuses on data as a whole, providing features like aggregation, joins, windows, etc. — README
@josevalim I understand that there are two Broadway pipelines, but this is out of the scope of why Flow, so let’s assume we have one for simplicity.
We have a complicated Flow
with several emits and we need to keep and pass an accumulator through the whole Flow. Basically, this is a processing of the huge CSV having several different partitions and a complicated logic to produce different new Ecto schemas out of it. Either I did not get how Broadway could be used instead, or we are stick to Flow for this particular processing. Partitioning inside these Flows is done by, say, properties.
OTOH, we want this Flows to act as plugins agnostic to the real data source. It might be several CSVs to be glued into one, or even many API calls with “a single row” data in each of them. For the latter, we want to use Broadway, processing different data sources and partitioning them, say, by client. This partitioning is relatively easy and does not require Flow.Window
and other sophisticated stuff from Flow.
And we want these parts to be independent as much as possible. I hope the above answers the question.
from broadway.
Related Issues (20)
- How to stop a Broadway Kafka pipeline? HOT 1
- Make producer module a keyword list to ease configuration management? HOT 6
- Broadway.update_rate_limit doesn't reset the counter/interval right away HOT 3
- NoopAcknowledger fails with ack key being set HOT 7
- Disable automatic call to handle_batch/4 HOT 2
- [Question] Creating a Broadway Message struct for testing?
- Telemetry distinguish between Producer metrics HOT 2
- Expected Behavior on Startup? HOT 4
- [docs] The `Broadway.test_batch` example doesn't work with Broadway 1.0.3 HOT 3
- Dialyzer error on ack_immediately/1 HOT 3
- Broadway v1.0.4 Broadway.NoopAcknowledger returns NoopAcknowledger instead of Broadway.NoopAcknowledger HOT 1
- Add `terminate/3` callback HOT 11
- Oban producer HOT 3
- Allow support for Nimble Options 1.0 HOT 5
- Request for MQTT support in Elixir Broadway HOT 3
- Issues using Broadway with DynamicSupervisor
- `handle_message` timeout HOT 2
- C
- Batcher Concurrency based on batcher_key HOT 2
- GenStage 1.2.0 breaks tests HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from broadway.