illinois-cs241 / broadway Goto Github PK

When the grader dies while running a job, that job should be marked as failed. Which does not seem to be the case as shown below:

This is when we need to maintain some graders or test some graders. So we don't want the job to be assigned randomly.

Following up from illinois-cs241/broadway-api#58, some warnings like shadowing variable names, type mismatch warning slip by flake8. We should be able to catch such things and not let them slip into production code.

Maybe change the linter? But have been unsuccessful in finding linters which can do this.

Example:

1. Course A requests 400 grading runs.
2. Course B requests 200 grading runs 5 minutes later.

Currently, no runs from Course B will run until all of Course A's runs have finished.

Assuming and giving a once over of a distributed system is good -- actually verifying it is a little better.

We don't need to do this immediately, but to be a robust piece of software, we should put in some Jepsen Tests for fault tolerance. We'll need to assume just a few likely cases (we aren't trying to make this a true fault tolerant distributed system),

• (this) API Fails
• Graders Start Failing in
  • A few at a time
  • Waves
• Various network delays cause graders to come back online (little less likely).

Right now there is essentially a single file that has all of the functions needed to start up the grader and run it.

I would suggest we make a Grader class that contains any grading logic and that we put it inside of the grader module. Then at the top level (outside of the module) we have a run.py that instantiates the grader and gets things running.

I think this would make things a bit easier to maintain.

Since we have containerized broadway API and grader, it would be a natural step to migrate the entire cluster to docker swarm (or other container orchestration system).

This is mostly an issue of configuration. There are a few things to solve in my limited experience with docker swam:

• broadway grader needs to interact with a docker daemon. dind doesn't work in this case since there are some permission issues (docker swam doesn't allow privileged container).
• broadway API need a mongodb instance. So we also need to figure out how to expose a mongodb service to the containers in docker swarm.

A few potential benefits in doing this:

• auto-deployment
• more fault-tolerancy
• easier to monitor the status of the entire cluster (?)

Since broadway-api and broadway-grader share some common parts and most changes usually require changing both repos, putting them in one repo might be more friendly to both maintainers and users.

Also as @ayushr2 mentioned, bundling them together and start tagging releases may help avoid compatibility issues in the future.

The problem is in the websocket handler

It's creating a completely new model.WorkerNode every time which resets a bunch of fields to their initial values. The correct behavior should be first checking if there exists one node with the given id then only update altered fields (hostname, etc.)

Need to change some stuff here https://github.com/illinois-cs241/broadway/blob/master/broadway/api/callbacks/worker.py#L50
to load-balance jobs between graders

Following discussion from illinois-cs241/broadway-api#61, we need to change how registration works. See this for implementation details.

We should be taking in the worker ID as a positional command line argument.

Currently, the tests do not test the following:

• The DB contents after jobs have been scheduled/executed or after courses are uploaded.
• Worker node failure. We will have to mock the variable which decided how often the heartbeat validator runs and we can make the testing thread to sleep (after polling a job) and then check DB contents to assert that the job is marked as failed.
• Worker node state (alive/dead)

Right now you can deploy the api after knowing a good amount about the internals, setting up mongo etc.

We have codified setup in .travis.yml, but it'd be nice to refactor this to a deploy script for Ubuntu, so we can

• Use this for travis testing
• Provision machines with a one step install

I'd recommend fab, but as always there are other ways to skin a cat.

We should test that when a node reconnects, their info is sustained and they can continue from where they left.

As usual, running a piece of code as root unless absolutely necessary is just good security practice. Can we have a script that creates a new user and sets up that user to be a part of the docker group instead of root?

The CI can be mostly copied over from broadway-api, but it'd be nice to have some assurance checks on each commit.