Source code of the JobAdder project
distributedtaskscheduling / jobadder Goto Github PK
View Code? Open in Web Editor NEWSource code of the JobAdder project
License: GNU General Public License v3.0
Source code of the JobAdder project
License: GNU General Public License v3.0
Setup & Start server
Add stop command to the command handler & verify it works
Systemd integration (service file, with sql deps and graceful stop)
Installation of the above (maybe a script?)
Configuration for the server.
Contains things like a list of special resources (different licenses and how many of them are available), and some options for the scheduling algorithm.
I think I was assigned these two in our last meeting.
Two questions:
This class sends emails when a job is completed.
The real proxy the Server uses for communication with the Worker.
We'll most probably need to have dummies here for the unit tests, so the WorkerServerProxy, etc. need to be interfaces.
Implementation of the dispatcher class.
Implementation of the Web server and all the associated requests.
Messages which the UserClient sends to the Server.
They operate on the ServerDatabase.
from ja.user.message
:
add
cancel
query
A mock implementation for the Proxy which the Worker uses to communicate with the Server. This class can then be used to test that for ex. the DockerEvents class correctly communicates with the Server that a job has finished/crashed, etc.
This is a database which is used for unit tests - it doesn't need to connect to a real database, but should implement the ServerDatabase
interface.
I don't think we have this anywhere.
As discussed in the meting, I think we should have only a single Response class instead of Response, ServerResponse, and WorkerResponse.
Since we currently have a lot of PRs open this change should wait until we have a lower amount of PRs open.
Uses #78
Class which starts all 3 components.
Messages which the worker sends to the server. They operate on the Database.
crashed
done
register
retire
The real proxy which is used by the user to communicate to the server.
If the connection to a work machine suddenly drops, we should eventually get a timeout on the SSH connection (or we may want to periodically send pings). Anyway, when this happens, it would make sense to mark the machine as offline and probably mark the jobs on it as CRASHED
.
Would be nice to have.
A mock implementation for the proxy that the Server uses to communicate to Workers.
The proxy which represents the Server on the Worker.
Related to #61
These are used by the default algorithm to distribute jobs among work machines.
Messages which the Server sends to the server.
Basically all the classes in ja.common.message.worker_commands
:
cancel_job
get_workload
pause_job
set_filters
start_job
We really should have a consistent style. My proposal is to use the official Python styleguide: PEP 8.There are also good tools which check this, for example flake8.
Another thing is checking the type hints consistency with MyPy.
Anything else we want to check?
Worker configuration classes, it should be able to read for ex. the IP address of the server.
The main class for the Server.
Should start all components and run the main loop.
Currently, its layout is suboptimal.
It should support multiple verbosity levels.
I added an argument of type ja.common.database.database.ServerDatabase
to the execute method of ja.common.message.server.ServerCommand
.
As a consequence common is now importing from server.
We had agreed that user, server, and worker should only import from common.
Should we allow that common also (rarely) imports from user, server, or worker?
Otherwise we would have to move the entire database into common.
Class which does the actual message transmission and calls the Remote on the remote machine.
The proxy which the user client uses to communicate with the server.
This is just about a mock implementation, which doesn't really send the commands over the network, but is used only in unit tests.
I just noticed we never set the type_name
which the command handlers use to differentiate between commands:
As a result, the server and the worker will never be able to parse commands (except if I am mistaken and this is set somewhere, but I could not find any reference to type_name
aside from the command handlers). I think we can either
Make the proxies pass it as an additional argument to the SSH connection
Make the SSH connection infer the type by looking at the Python class name (object.__class__.__name__
)
What do you think is the better approach? My personal preference is 2. since it is easier to implement and won't require changing the interfaces, plus removes the need to manually specify the type everywhere.
The real implementation of the database class which will be used in production.
Implementation of the Remote program. Not sure how to unit test this properly, but probably we can extract the Job owner verification and unit test that.
The handling of command line arguments by the user client.
A mock implementation for the ProxyFactory, which returns mock WorkerProxies
The main class for the worker, using a mock server proxy from #64
Should also listen on the local socket for new commands from the server (that part could probably be shared with the server, should be investigated)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.