Comments (17)
I've implemented this on a separate branch connect_auto_chan_creation
, please give it a try.
This method has a downside though: it is not possible to set buffer capacity per channel, you can only set the default one for all connections:
flow.DefaultBufferSize = 1
(The default value set in the flow package itself is 32)
It would be nice to pass the buffer capacity as an optional argument to the Connect() function but Go doesn't support optional arguments. So it needs some kind of workaround.
from goflow.
Cool! Will test out ... but yeah, as you say, it would be quite desirable with per-channel buffer capacities, as it seems one can often tune the performance of the system quite a bit with those.
from goflow.
Works flawlessly, AFAIS!
from goflow.
As there are no default parameter values in Go, we should probably use 2 different functions to create a connection with a default buffer size or with custom one:
net.Connect("src", "srcPort", "dst", "dstPort")
net.ConnectBuf("src", "srcPort", "dst", "dstPort", 32)
The question is: should the channels be buffered or non-buffered by default and what default buffer size is fine. If we use async processes by default, then probably non-buffered channels should be fine and lighter on memory footprint.
from goflow.
+1 for that. Was thinking about the same (a custom method), but didn't have any good naming suggestion. ConnectBuf sounds good, I think.
I also agree with non-buffered channels as default, as it probably gives the least surprise to newcomers ... If there's not a too big performance hit with using unbuffered channels, that is...
from goflow.
The main problem with unbuffered channels is that it makes synchronous networks need to run in lockstep, which probably decreases overall performance quite a bit.
But as you say, if async processes are the default, it is probably better with non-buffered channels anyway, complemented with clear documentation on the recommended combination of async/sync with buffered/unbuffered channels.
from goflow.
In just a few words, buffer tweaks vs. performance is tricky, it depends on throughput of each node in the network. In actor-like async mode it is even more tricky because goroutine stack acts as a kind of buffer of its own.
from goflow.
In just a few words, buffer tweaks vs. performance is tricky,
it depends on throughput of each node in the network
Yeah, I can imagine that is the case (and actually I had the same results, from playing around so far).
In actor-like async mode it is even more tricky because
goroutine stack acts as a kind of buffer of its own.
Aha, right!
from goflow.
This feature has now been merged into the master branch. This means that dependent code should be updated to either use Graph.Connect() with just 4 arguments or use Graph.ConnectBuf() instead.
from goflow.
It would be nice to pass the buffer capacity as an optional argument to the Connect() function but Go doesn't
support optional arguments. So it needs some kind of workaround.
I happened to need this and a search in the std lib http://golang.org/search?q=Options found some ways of doing it (kind of) see http://golang.org/pkg/net/http/cookiejar/#New or http://golang.org/pkg/image/jpeg/#Encode .
Maybe it's what you are looking for. if not sorry for the noise.
thanks for your work on goflow.
from goflow.
Thanks for pointing to that. Yes, passing a struct of options is the Go way around optional arguments. But in this specific case, is
net.Connect("src", "srcPort", "dst", "dstPort", nil)
net.Connect("src", "srcPort", "dst", "dstPort", &flow.ConnectOptions{BufferSize: 32})
better than
net.Connect("src", "srcPort", "dst", "dstPort")
net.ConnectBuf("src", "srcPort", "dst", "dstPort", 32)
?
from goflow.
In personal opinion, I tend to think that the latter option (net.ConnectBuf) will be more self-documenting and straight-forward (imagining how this will look in godoc), if there are not any really concrete benefits of the former?
from goflow.
In short NO .
For me the first code is more explicit and the second is a bit more implicit and I look at the doc to know.
I do need to look at the doc for the other arguments so the route you took is the way to go the doc is better.
from goflow.
Well, I should admit that I'm not really experienced in library design, so I should probably better shut up anyway, and leave the decision to others :)
from goflow.
The benefit of the 1st approach is using more than 1 optional parameter, e.g.
net.Connect("src", "srcPort", "dst", "dstPort", &flow.ConnectOptions{BufferSize: 32, Name: "Some connection"})
I have a feeling that the library API will change several times along the way until we reach something "stable". For now I'd stick with ConnectBuf(), but if we add more parameters later, then it will be worth reconsidering the use of options struct.
from goflow.
Well, I should admit that I'm not really experienced in library design, so I should probably better shut up anyway :)
@samuell Well, for me I'll admit that I'm not at all experienced in coding, so I should shut up right now :) I'm just learning every day a bit more.
@trustmaster thanks again
from goflow.
Well, it is nice that we get this functionality, in any way! :) Keep up the good work, @trustmaster !
from goflow.
Related Issues (20)
- What is the current state of this project? HOT 7
- Tests failing with Go 1.7+ HOT 1
- Multiple Inputs connecting to same Output HOT 2
- Component with multiple inputs not stopped HOT 4
- Feature request: Broadcast from one proc port to n proc ports HOT 3
- Handlers on a Looper are never called HOT 1
- Components with private channels cause a panic
- Races, lots of races HOT 4
- dead links in the README HOT 1
- Routing
- Some code makes me confused HOT 1
- Feedback needed HOT 9
- v1 Design proposal
- flow performance HOT 1
- TestInternalConnectionIIP test panics randomly: IIP write to a closed channel HOT 9
- Closing Component channels that are not explicitly marked as ports HOT 5
- Manipulate a network in running
- cargo run readyset-server error HOT 1
- Will this repository continue to be updated? HOT 1
- Supporting multiple IIPs in one network?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goflow.