psiayn / goheiko Goto Github PK
View Code? Open in Web Editor NEWHeiko reinvented in go!
Heiko reinvented in go!
When all nodes are down (or you have only one node and it's down), heiko keeps trying to connect repeatedly. This leads to high CPU usage (6-10% out of 800% on my laptop, which isn't very high but is abnormal).
heiko should slow down trying to connect to a node when connections repeatedly keep breaking, have some timeout between connections.
One idea I could think of:
One improvement to this would be to store the last 3 (or 5, or N) failures, increase the sleep time as more failures occur.
This puts the responsibility of choosing an appropriate node on the scheduler. If it chooses a node that fails constantly, it will lead to delayed execution.
Copying of keys ( if not done already ) should probably be done in the init phase.
PID File generated in daemon mode is supposed to be deleted via the context.Release()
call. As of right now, any form of interrupt ( SIGTERM / SIGINT ) doesn't seem to change it's behaviour.
Also go-daemon seems to be pretty old, could that be the issue?
Heiko allows a custom SSH private key to be used for auth, instead of the one generated by heiko. This can be set in the configuration as shown:
https://github.com/psiayn/heiko/blob/fb8b464ec2d4308fde9a4d6cfbcc2762d2851783/examples/sample-config.yml#L9
Generally, these keys are stored in ~/.ssh
(which is $HOME/.ssh
). Providing the path with the ~
(tilde) in it is not recognized by heiko, it exits with this error:
init: SSH Key ~/.ssh/somekey.pub for node somenode does not exist: stat ~/.ssh/somekey.pub: no such file or directory
When the paths are read on this line:
https://github.com/psiayn/heiko/blob/fb8b464ec2d4308fde9a4d6cfbcc2762d2851783/internal/config/sshSetup.go#L110-L112
If it starts with ~/
, it must be expanded appropriately. Possible approaches for this are here and here.
As of right now init command only runs commands specifies in the config file.
Init should in theory do the following:
A scheduler in heiko is the part that chooses which node to run a given task on. Tasks are sent over a task channel. The scheduler receives tasks over this channel and runs that task on a node (in a new goroutine).
Currently (as of fb8b464) heiko has just one scheduler - a random scheduler that selects a node at random.
A few ideas for new schedulers:
Before implementing new schedulers it would be a good idea to have an interface for a scheduler. If you look at the random scheduler, only one line of it actually does the job of choosing a node to run the task on: https://github.com/psiayn/heiko/blob/fb8b464ec2d4308fde9a4d6cfbcc2762d2851783/internal/scheduler/scheduler.go#L41
Everything else is boilerplate that will have to be repeated if a new scheduler is added with the current code structure.
Tasks to complete before (or along with) implementing a new scheduler:
cmd/
, uses cobra) to select a schedulerCurrently, all of the heiko configuration is one file .heiko/config.yml
. This includes both nodes and jobs. This makes it difficult for the config to be shared (in a git repository for example) without exposing details of the user's nodes which can possible include passwords. The idea is this
.heiko
directory (like here) that only has details of what tasks are needed to run this project using heiko. Similar to a Dockerfile
or docker-compose.yml
. Having a .heiko
directory in different projects also increases exposure to heiko (to some extent).Break up config.yml
into two files jobs.yml
and nodes.yml
.
jobs.yml
will live in .heiko/jobs.yml
and be local to each project. It only contains the jobs
part of the configuration. This allows it to be added to version control.nodes.yml
can be in .heiko/nodes.yml
(local to the project, but not checked into version control) or if that does not exist, the user's global ~/.heiko/nodes.yml
will be used. This way, the same nodes can be used for multiple projects.Some possible issues:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.