jcarreira / cirrus Goto Github PK

#52

It seems sometimes the travis tests fail because not enough time is given for the test to converge.

Should we increase the threshold? @andrewmzhang @yangryan0

There are issues with the python frontend when the plotly version is 3.1.0. The plotly version has to be exactly 2.7.0 to work.

The 0th poll thread needs to start last. We need a barrier to ensure this, or else for large numbers of poll threads, parameter server is likely to fail.

It seems that a few people (at least me, Andrew and Neel) have found that travis clang-format sometimes suggests different code formattings than the one installed locally.

I'm opening this issue to advance towards a solution to this problem.

This bucket is not exist now,so I want to creat it again. But I'm confused what is in it？Looking forward to ur reply~~

SKLearn model objects provide a predict function that lets the user provide input for predictions/evaluations on the trained model. Cirrus should do that as well.

Periodic checkpointing of model to remote storage. #19

New softmax model for Cirrus

These tests should be used to test the system in different environments: Windows, OSx.

If I start a grid search run with UI on and kill the experiment mid way, it seems I can't rerun it without restarting jupyter.

The reason seems to be that the dash server is still bound on the port. What is the solution for this?

S3.cpp needs to be refactored to a more object-oriented interface. As of now, this module has a couple of issues:

1. no way to create Aws::Client's with different options
2. no easy way to pass clients to S3 without passing these AWS::Client handles

S3.cpp should be a class (I suggest S3Client) so users can create multiple of these clients.

Is there any reason that there're two lambda definition exists in the code base?
One is

Right now we read the tail of the log file from the ErrorSparseTask.

We should probably create an UDP/TCP interface in the error task,

In one file:

pip install cirrus
#59

Define which formats Cirrus supports

Currently, our experiments only terminate when the user calls stop in the python interface. We should have an option to set a limit, so that the experiment automatically stops when it reaches a certain accuracy or some other limit.

As of now we don't count 1) cost of bandwidth from S3 to the Cirrus workers, 2) cost of S3 requests.

The cost of requests can be expensive for very high IOPS.

If user's public key is already in the server's .ssh/authorized_keys file, there is no way to specify an empty .pem file.

Make a list here of different ML algorithms we could support on Cirrus.

Add support for Latent Dirichlet Allocation on Cirrus
#51

Run multiple workloads at the same time and show progress in realtime with multiple plots.

#49

If the Error Task requests a model after the parameter server has started but before the PS has initialized the model, the error task will segfault.

Fix bug in min max scaling in python tests

We should leverage flatbuffers for building messages used in the communication between the workers and the parameter server.

We should add a .clang-format to run in travis.

Consistent with old bug reported in multiple places, including here: https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Need to shard the models across multiple VMs/lambdas to provide more aggregate bandwidth.

https://docs.google.com/document/d/1G-2HCMcUWVcRAGICb5u0_b8HYcP66uTWEvGnWK1rU5A/edit?usp=sharing