Coder Social home page Coder Social logo

trendingtechnology / fiber-1 Goto Github PK

View Code? Open in Web Editor NEW

This project forked from uber/fiber

0.0 2.0 0.0 8.61 MB

Distributed Computing for AI Made Simple

Home Page: https://uber.github.io/fiber/

License: Apache License 2.0

Dockerfile 0.05% Makefile 0.17% Python 84.16% Shell 0.07% HTML 7.25% CSS 4.56% JavaScript 3.75%

fiber-1's Introduction

build

drawing

Project Home   Blog   Documents   Paper   Media Coverage

Join Fiber users email list [email protected]

Fiber

Distributed Computing for AI Made Simple

This project is experimental and the APIs are not considered stable.

Fiber is a Python distributed computing library for modern computer clusters.

  • It is easy to use. Fiber allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.
  • It is easy to learn. Fiber provides the same API as Python's standard multiprocessing library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.
  • It is fast. Fiber's communication backbone is built on top of Nanomsg which is a high-performance asynchronous messaging library to allow fast and reliable communication.
  • It doesn't need deployment. You run it as the same way as running a normal application on a computer cluster and Fiber handles the rest for you.
  • It it reliable. Fiber has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.

Originally, it was developed to power large scale parallel scientific computation projects like POET and it has been used to power similar projects within Uber.

Installation

pip install fiber

Check here for details.

Quick Start

Hello Fiber

To use Fiber, simply import it in your code and it works very similar to multiprocessing.

import fiber

if __name__ == '__main__':
    fiber.Process(target=print, args=('Hello, Fiber!',)).start()

Note that if __name__ == '__main__': is necessary because Fiber uses spawn method to start new processes. Check here for details.

Let's take look at another more complex example:

Estimating Pi

import fiber
import random

@fiber.meta(cpu=1)
def inside(p):
    x, y = random.random(), random.random()
    return x * x + y * y < 1

def main():
    NUM_SAMPLES = int(1e6)
    pool = fiber.Pool(processes=4)
    count = sum(pool.map(inside, range(0, NUM_SAMPLES)))
    print("Pi is roughly {}".format(4.0 * count / NUM_SAMPLES))

if __name__ == '__main__':
    main()

Fiber implements most of multiprocessing's API including Process, SimpleQueue, Pool, Pipe, Manager and it has its own extension to the multiprocessing's API to make it easy to compose large scale distributed applications. For the detailed API guild, check out here.

Running on a Kubernetes cluster

Fiber also has native support for computer clusters. To run the above example on Kubernetes, fiber provided a convenient command line tool to manage the workflow.

Assume you have a working docker environment locally and have finished configuring Google Cloud SDK. Both gcloud and kubectl are available locally. Then you can start by writing a Dockerfile which describes the running environment. An example Dockerfile looks like this:

# example.docker
FROM python:3.6-buster
ADD examples/pi_estimation.py /root/pi_estimation.py
RUN pip install fiber

Build an image and launch your job

fiber run -a python3 /root/pi_estimation.py

This command will look for local Dockerfile and build a docker image and push it to your Google Container Registry . It then launches the main job which contains your code and runs the command python3 /root/pi_estimation.py inside your job. Once the main job is running, it will start 4 subsequent jobs on the cluster and each of them is a Pool worker.

Supported platforms

  • Operating system: Linux
  • Python: 3.6+
  • Supported cluster management systems:
    • Kubernetes (Tested with Google Kubernetes Engine on Google cloud)

We are interested in supporting other cluster management systems like Slurm, if you want to contribute to it please let us know.

Check here for details.

Documentation

The documentation, including method/API references, can be found here.

Testing

Install test dependencies. You'll also need to make sure docker is available on the testing machine.

$ pip install -e .[test]

Run tests

$ make test

Contributing

Please read our code of conduct before you contribute! You can find details for submitting pull requests in the CONTRIBUTING.md file. Issue template.

Versioning

We document versions and changes in our changelog - see the CHANGELOG.md file for details.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Cite Fiber

@misc{zhi2020fiber,
    title={Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods},
    author={Jiale Zhi and Rui Wang and Jeff Clune and Kenneth O. Stanley},
    year={2020},
    eprint={2003.11164},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

Acknowledgments

  • Special thanks to Piero Molino for designing the logo for Fiber

fiber-1's People

Contributors

agrimrules avatar calio avatar danilopeixoto avatar jimjag avatar rongou avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.