Comments (4)
Ex:
Cost of putting things into tuples
Cost of selecting Nth value as the join key (rather than only the first)
New join_by
operator?
from hydroflow.
Can maybe tell just by the current flame graphs if this is causing any performance overhead? @zzlk
from hydroflow.
I took a quick initial swing at this and from what I can tell, to my surprise, the behavior of the mapped versions do indeed perform differently. I was not expecting this. I was expecting the mapped ones, as long as the data is the same, to compile down to exactly the same code, so no difference should be detectable. I did try to create some scenarios in https://godbolt.org/ but it is hard to coax the compiler into doing what you want, and then interpreting the resulting assembly is harder still. This would normally be the approach I would take with C/C++ but for rust it didn't work out. So instead I went with the benchmarking approach.
The benchmark I made was to feed data into a join, and time how long it takes to do the whole join. The data that is generated is either fitting the join or needs a map, these two options are compared against each other. A similar thing is done for the output side, giving 4 total combinations [(nomap, nomap), (map, nomap), (nomap, map), (map, map)]. I also added one more test case where the output map outputs a larger tuple than what is inputted, it duplicates one of the tuple elements. This is a control, it should be slower, since it is copying memory/outputting more data.
I ran the benchmarks with
sudo nice -n-20 taskset 0x4 sudo -u stan cargo bench -p benches -- mapjoinperf --output-format bencher --measurement-time 10
making sure to disable the intel turbo boost feature on my cpu otherwise it's hard to get sensible results:
echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
and I got these results:
So, likely need to spend a bit more time on this, make the key-access operator and see how it performs in comparison to map in/map out, as well as try to understand the above benchmark, why adding a map() can sometimes make it faster.
from hydroflow.
Hypothesis is that LTO makes this irrelevant; need to confirm.
from hydroflow.
Related Issues (20)
- Make sure that stageleft can handle `super` and `crate` references
- Rohit Onboarding Projects
- `surface_codegen.rs` `test_surface_syntax_reachability_generated` needs `'static` for join
- Stageleft sometimes double-captures a referenced value
- Documentation Improvements to the Networking Examples HOT 1
- Improvements to Hydroflow Template HOT 1
- Hydroflow surface syntax allows ports specification that isn't used `[0]out -> in`
- Deprecate/remove hydroflow surface syntax `import!("file.hf")` feature
- Hydroflow surface modules (`import!("file.hf")`) panics without input/output `mod` HOT 1
- persist should be lattice typed, e.g. to enforce uniqueness of sets HOT 2
- Expected a defer_tick to be required, but isn't HOT 5
- Create CONTRIBUTING.md with info for new developers HOT 3
- Properties/semantics/optimizations meta-issue (algebraic properties/lattices) HOT 1
- Stageleft doesn't allow `let _`
- Check & Update Client-Server Examples Based On Updated Template
- Running with `--profile profile` crashes with "Illegal instruction (core dumped)" HOT 2
- Support `let (a, b) = ...` in stageleft
- Generate code for locations even if they have no sinks
- Print a warning when a stream is dropped without being used by a sink HOT 1
- Sending a locally-defined struct with bincode results in bad references in the generated code
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hydroflow.