Coder Social home page Coder Social logo

Multiple processes and threads? about bionic HOT 1 OPEN

square avatar square commented on July 25, 2024
Multiple processes and threads?

from bionic.

Comments (1)

jqmp avatar jqmp commented on July 25, 2024

Sure, that's a great idea. I'll do that this week. Let me run a short version by you to make sure it actually answers your question:

  • It's probably not safe to run the same flow simultaneously from multiple threads.
  • It's probably safe to run the same flow simultaneously from multiple processes. It's not really the intended usage but we've tried to design the cache in a way that makes this safe. However, this may lead to duplicated work, such as multiple processes starting the same expensive computation and then each caching their own instance of the output.
  • It's probably safe to run the same flow simultaneously on multiple hosts. I know people at Square are doing this now (for distributed hyperparameter tuning) and they haven't reported any concurrency bugs, although they do get annoyed by the duplicate-work issue above.

I'd love to hear more about the use cases you have in mind. Ultimately we'd like to have Bionic handle this for you inside a single flow -- and in the next 48 hours we should be releasing 0.8.0, which will be able to compute entities in parallel (separate processes on one host).

from bionic.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.