gojek / proctor

A Developer-Friendly Automation Orchestrator

GO-JEK Technologies

Description

Middleman-Powered GO-JEK Technologies's Website 👻 🕸 hosted with ❤️ by GitHub

GO-JEK Technologies generates matter for GO-JEK Technologies on GitHub Pages

Building

try, 
  make clean
  make setup
  make install
  make serve

or just,
  make

License

Copyright 2018, GO-JEK Technologies <http://www.gojek.io>

proctor's Issues

Enable auto-completion in bash for basic commands

We can have auto-completion for describe/execute/list commands.

• Out of scope:

auto-completion for procs and their args.

Encrypt secrets while storing them

As an admin,
I want to encrypt secrets while storing them using proctord.

So that,
If secrets get leaked accidentally, they aren't of any use without the encryption key.

Remove the redundant `execute` command for proc execution

As an admin,
I don't want users to type execute everytime they want to execute a proc.

Current scenario:

proctor execute create-some-infra-one ARG_ONE=a ARG_TWO=b
proctor execute create-some-infra-two ARG_THREE=c ARG_FOUR=d

Better scenario:

proctor create-some-infra-one ARG_ONE=a ARG_TWO=b
proctor create-some-infra-two ARG_THREE=c ARG_FOUR=d

This will also help move towards resolving #32

Remove proc command from proctor CLI

As a user,
I don't want to type the redundant word proc everytime I list/describe/execute procs.

Load config only once per run

• Move InitConfig to main.go
• Don't make config as Singleton
• Clean-up the existing test for configs

Notify CLI users when newer version of proctor is released

As an admin,
I want to notify CLI users when a newer version of proctor is released.

So that,
Users can stay updated with the latest features.

When a newer version of proctor (compared to the installed version) is released, Users should see a notification on every command run from CLI. (Example: gcloud/aws CLIs)

Add Makefile for proctor-cli

• Add Makefile for proctor-cli

Test proctor client config setup command

As an admin,
I want to test logic written inside config setup.

So that,
On modifying it in future, existing functionality doesn't break.

[Bug] Fix job execution status updation logic

In kubeclient, on job failure, it doesn't return the status immediately and keeps on watching for more events

• The event watching channel isn't being closed and can lead to file descriptor leaks.
• Add an updated at column in database when updating a record.

Track job execution status

As an admin,
I want to track which jobs submitted for execution succeeded and failed.

Every job which is successfully submitted to proctor, will be running as a script. The job success/failed will depend on exit code of the script. If exit code is 0, the job ran successfully. Else, failed.

Namespace config of CLI under proctor

As an admin,
I don't want existing config of users to be accidentally used when running proctor. Currently, CLI has config with keys EMAIL_ID and ACCESS_TOKEN. Since these are common keys, user might have them set as ENV vars.

When proctor runs, it might pick preset config without user's knowledge.

In order to avoid such behavior, we can namespace config under PROCTOR_. Example: PROCTOR_EMAIL_ID and PROCTOR_ACCESS_TOKEN

Proctor CLI exits with non-zero status on proc failure

As an admin,
I want the proctor CLI to exit with non-zero status when a proc fails.

So that,
They can use proctor CLI inside their scripts and have proper feedback on it's success/failure.

Technical details:
Post logs streaming is completed, poll proctor daemon for the status of proc until it's successful/failed.

Refactor auditing of proctor jobs

Currently,
Golang's context is used to pass auditing metadata. This is a bad practice.

Instead,
We could have a builder object and pass it for auditing at the end. That will be more cleaner.

Getting status of a proc submitted to proctor service

POST /jobs/execute returns a job-id to the user. But there is no way to get status of a job submitted to service.

Let's expose an API to fetch the status of a job.

List procs in alphabetical order

When showing the list of procs, for enhancing UX. It would be nice to have the list of the procs to be shown in an alphabetical order.

Provide free-flowing args while executing a proc

Currently, for providing any args to a proc, I need to pass them as K-V pairs.
Example:

proctor execute grep-knife-node-list NODE_NAME=proctor-postgres

Would it make sense to pass free-flowing args while executing a proc?

Example:

proctor execute grep-knife-node-list proctor-postgres proctor-redis

Details:
proctord can provide all the free flowing args as a default(convention) env var to a proc
The proc will have logic to parse the args from default env.

Move dependency management to go modules

As of go 1.11, modules is the official dependency management tool in golang.

It makes sense to move proctor to use modules. Changes that might be required:

1. Change the directory structure from GOPATH to outside GOPATH
2. Have one dependency manager for both proctor CLI and proctord (currently we have 2 glide.yaml files)
3. Have one binary for CLI and proctord

Have proctor CLI update itself

As an admin,

Post being notified from the CLI on a new release of Proctor CLI, users can update proctor from the CLI itself. Example: proctor update <version(optional)>

Notification on Proc completion

As a user,
I want to be notified when the proc I executed completes.

So that I don't have to pay attention to the progress of procs in execution, and can focus on something else.

Once a proc reaches completion, proctor should notify me about that, and let me know if the proc completed successfully or failed.

Out of scope:
Multiple channels of notification.
For now, proctor will only accept an api during executing a proc. It will notify on that api once the proc reaches completion.
Implement multiple channels for notification later.

Fix discrepancy between jobs/procs/automation

As an admin,
I want to clean-up inter-changeable usage of the keywords jobs, procs, scripts and automation inside Proctor. Across Proctor, only 1 keyword should be used for the automation run.

So that,
A user isn't confused. Usage of only one keyword will help achieve clarity in the mind of users using Proctor.

Run procs in specified environment

As an admin,
I want users to be able run procs in the environment they specify.

So that,
Connectivity issues won't arise while running procs.

Currently procs are run in a single predefined environment. Hence, users need to ensure connectivity to separate environments. This feature will increase the capabilities of procs and reduce the headache of maintaining connectivity with multiple environments.

Retry logs streaming from CLI on failure

Currently, CLI (client) doesn't retry on failure while streaming logs. This removes the ability to deploy proctord at desirable hour.

Add Proctor Api Documentation

Alternative methods to specify input parameters for procs

When input parameters for a proc are large, it sometimes becomes difficult to verify the input params one last time before executing a proc. Does it make sense to add a feature to the CLI to read input params from a JSON file?

Throws error on virgin proctor config command invocation

Should automatically create whatever directory & write configuration files it needs. I do not understand why does it throw this error.

Incorrect error messages on proctor CLI on proctord failure

When an authorized user executes a proc,
If proctord responds with a non-successful response,
The user is shown an error: Error executing proc. Please check configuration and network connectivity
Instead the error message should read: Proc execution failed. Please consult admin.

Add Email-Id and Access-Token Headers for authentication

As an admin,
I want users to send Email-Id and Access token when interacting with proctorD from proctor-CLI.

So that, I can audit users' name on execution of procs, submission of secrets, etc.

Add init command that bootstrap proctor config

Add proctor init command for first time user to generate proctor config file based on parameters.

Procs are rerun on failure

As an admin,
I want to configure the number of retries for procs on failures.

So that,
Users contributing to procs can have control if they haven't handled immutability, state, etc. A proc will be run once under all circumstances. On failure, it can be configured to not rerun.

Currently, it's retried 6 times on failure by default.

Stream logs from api-server directly

• Currently kube proxy is used for streaming logs
• This will reduce one point of failure for proctor

Incorrect doc for proctord config

In code it expects: PROCTOR_KUBE_BASIC_AUTH_ENCODED as key of kube basic auth.

In doc it mentions: PROCTOR_KUBE_BASIC_AUTH

Intuitive help for procs

As a user,
I want procs help to be more intuitive.

Currently, If I fetch all procs using proctor list, and I need help for run-sample proc, I need to type command: proctor describe run-sample. That's unintuitive.

Instead, proctor help run-sample or proctor run-sample --help would be more intuitive for displaying help.

Persist fetching the procs execution status job

As an admin,
I want to persist the procs execution status job for a worker to pick it up.

Currently, it is being run asynchronously using a goroutine. In that case, if there's a deployment of proctord, the proc execution status won't be updated in the database.

Timeouts on CLI when connecting to proctord

The CLI currently has infinite timeout when connecting to proctord. If there's connectivity issue, the user won't know. Instead have a sane timeout to fail fast and give user relevant feedback.

Validate .proctor.yaml configuration

0. Validate $HOME/.proctor/proctor.yml file exists, if not give instructions on user about creating the yaml files
1. Validation .proctor.yml file's presence and check for mandatory parameters like PROCTOR_HOST, EMAIL_ID and ACCESS_TOKEN. Give warning if parameters missing.
2. If received 401 from server, check for presence of EMAIL_ID and ACCESS_TOKEN and ask for setting the same
   2.1 If received 401, and EMAIL_ID and ACCESS_TOKEN present, give user warning on invalid ACCESS_TOKEN and regenerate one
3. If unable to reach host, print out error message for reachability and checking internet/vpn connection

Getting the issue while installing proctor.

github.com/spf13/cobra
gojek/src/github.com/spf13/cobra/command.go:31:25: undefined: pflag.ParseErrorsWhitelist
github.com/googleapis/gnostic/extensions
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:57:29: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:121:45: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:180:46: undefined: proto.InternalMessageInfo
gojek/src/github.com/googleapis/gnostic/extensions/extension.pb.go:237:29: undefined: proto.InternalMessageInfo
github.com/gojektech/proctor/proctord/storage
gojek/src/github.com/gojektech/proctor/proctord/storage/store.go:73:33: multiple-value uuid.NewV4() in single-value context

Notify users on new release of Proctor CLI

As an admin,
I want to notify CLI users when a newer version of proctor is released.

So that,
Users can stay updated with the latest features.

When a newer version of proctor (compared to the installed version) is released, Users should see a notification on every command run from CLI. (Example: gcloud/aws CLIs)

Not able to delete procs from proctor.

I have removed one job from my proctor job repo and I have updated the metadata without the removed job. But still I can able to see the job in the proctor proc list

Add meta fields author, contributor and organization to proctor scripts

Integrate newrelic instrumentation with proctor

I want to instrument the code, enable application-level metrics for proctord.

So that,
In the case of buggy behaviour, I can correctly diagnose the root cause of the problem.

Add readme for setting up automation orchestrator

As a user,
I would like to see how to stitch all components of Proctor together.

Currently, dev setup for proctor CLI and proctord exists.

What already exists:

• Setup proctord with a secrets store, procs engine and audit logging store
• Setup proctor CLI to talk to proctord

Acceptance criteria:

• Setup CI for packaging procs inside images and push to registry
• Setup CI for procs metadata to be populated in proctord

Notice to upgrade proctor CLI when out of date

As an admin,
I want to notify users when new version is available whenever they run proctor version

Example:

$terraform version
Terraform v0.11.7

Your version of Terraform is out of date! The latest version
is 0.11.10. You can update by downloading from www.terraform.io/downloads.html

Support for Multi Proctor profiles

Just like AWS CLI, it will be better if procs support multi-profile, so we can use proctor for more than one proctor service.

Currently, if we want to use proctor to other proctor services, we will need to modify proctor config file.

Don't display `Constants` when asked for help on a proc

As a user,
I mostly see help to run a proc. Since I don't have to provide constants to run a proc, it deteriorates my experience.

An admin can fetch the required constants to submit secrets for a proc from the procs metadata file. So, it makes sense to remove display of Constants from the CLI.

Incorrect error message when running jobs without pemissions

Proctor shows the following error message when I try to run procs that I don't have permissions for.
Error submitting proc for execution

It should show a better error message so the user knows why the proc failed.

Display authorized groups when describing a proc

As a user,
I want to see authorized groups for a proc when seeing the help for it.

Add uniqueness constraint on job names submitted for execution

As an admin,
I want to add uniqueness constraint on column job names submitted for execution.
Because:

• As of now, proctord has only one engine to run jobs. So there shouldn't be any conflicting names
• As number of records grow, the query to fetch jobs by execution names will become slower

Fix PROCTOR_URL behaviour

The PROCTOR_URL name of the config variable implies that it must be a URL. But in the code, it is used as the hostname. This is confusing.

Example lines:
https://github.com/gojektech/proctor/blob/master/daemon/client.go#L42
https://github.com/gojektech/proctor/blob/master/daemon/client.go#L64

Mandatory variable check without executing Procs

Currently, Proctor CLI will do variable check by executing procs on the executor.
Need to add a feature on CLI to check Mandatory Variable without contacting Proctor Service.

Schedule Procs

As a user,
I want to schedule procs to run at a defined time and date.

So that I don't have to manually execute the proc at that time. proctor gets executes the proc.

Out of scope:
Only schedule a proc to run once. For executing procs in a recurring schedule, implement later.