Comments (1)
Sure, that's a great idea. I'll do that this week. Let me run a short version by you to make sure it actually answers your question:
- It's probably not safe to run the same flow simultaneously from multiple threads.
- It's probably safe to run the same flow simultaneously from multiple processes. It's not really the intended usage but we've tried to design the cache in a way that makes this safe. However, this may lead to duplicated work, such as multiple processes starting the same expensive computation and then each caching their own instance of the output.
- It's probably safe to run the same flow simultaneously on multiple hosts. I know people at Square are doing this now (for distributed hyperparameter tuning) and they haven't reported any concurrency bugs, although they do get annoyed by the duplicate-work issue above.
I'd love to hear more about the use cases you have in mind. Ultimately we'd like to have Bionic handle this for you inside a single flow -- and in the next 48 hours we should be releasing 0.8.0, which will be able to compute entities in parallel (separate processes on one host).
from bionic.
Related Issues (20)
- Errors from unset entities are hard to debug
- @changes_per_run generates extra cache entries
- Non-persistable entities can sometimes be spuriously recomputed
- Redefining an entity with a function can produce surprising results
- The "provider" concept is too broad
- Should bionic.util.init_basic_logging only affect logging for bionic components?
- Concerning exception messages in pytest on Python 3.8 HOT 4
- Compute task states in the subprocesses
- Full test suite fails on OS X Catalina HOT 1
- Caching fails when directory names contain spaces
- `FlowBuilder.merge` not accepting another builder is confusing
- Reloading of flow spanning multiple files HOT 1
- Docs: Functions with multiple outputs cannot be called explicitly HOT 2
- Uncached values should be recomputed each time they're used HOT 1
- Set-valued entities are not hashed determistically
- Confusing error message when non-iterable input used with `outputs`
- parallel evaluation inconsistencies with persist settings
- Improve error message when protocols fail
- Test vector update script should delete old test vectors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bionic.