nathro / autotransform Goto Github PK

Continuous runs are automatically scheduled runs that can be used to keep a codebase cleaned-up/maintained over time. Runs could be scheduled hourly, daily, or weekly. The information for scheduling will be contained in a schema's Config.

The updates here will include:

• Renaming our existing schema configs to Params to distinguish from ConfigFetcher's
• A ContinuousSettings object that is an optional part of SchemaParams used to define when to trigger a run
• A cron.py script that will pull all existing SchemaBuilder's as well as any schemas located in data/schemas, check if they should run, and then run them. Parameters will be: -h (the hour of the day the run was scheduled for) -d (The day of the week the run was scheduled for) -w (The worker object to use)
• A section in CUSTOM_DEPLOYMENT describing utilizing the script

Future functionality:
While not part of the existing plan, in the future we may integrate this type of behavior with Github workflows/actions if possible to allow people using Github to leverage actions to automate this process for them on push/other events.

We can definitely remove the runner from each of these to simplify things, maybe more.

Some organizations may not allow you to merge directly to main, so we need an action that can trigger a rest API for things like merging.

Github Actions isn't the only game in town. For now, it looks like we'll need to use Jenkins for some of our repos at Slack so I'm going to add Jenkins support to AutoTransform.

Make GithubRepo object able to put in comments (including inline comments)

Add a system for logging to AutoTransform that has integration points for custom deployments to leverage logging.

This will be done through the following:

• Event: A base class for all AutoTransform events. Provides a message method that can be used for outputting to stdout.
• EventHandler: A handler that will take each event and use it to log to stdout. Can be extended in custom deployments with a log method to log elsewhere as well.
• LoggingLevel: A level which determines what gets displayed in logs. Will have 3 values: minimal, normal, and verbose.

We need to add a system whereby AutoTransform PRs can be automatically approved. Need to decide on the implementation for this such that it is both clean and secure.

Add some Github actions to the repo to make sure it stays in good health by adding actions for:

• mypy
• pytest
• pylint
  That all run on push. I'll also look at actions around building the package and deploying to pypi but that may not happen.

Right now Change has take_action() method, we should get rid of that and have the actions have a run() method instead. This will prevent needing the million if statements in take_action(). We'll still have methods on the Change for things like merge()/close()/etc...

Existing documentation is lacking a few key things that would make picking up and understanding AutoTransform easier.

This includes 3 new .md files:

• EXAMPLE: Runs the user through a toy example of creating and running a transformation
• DATA_FLOW: Explains the flow of data through AutoTransform and how the components all work together
• FAQ: A simple FAQ with a few questions for now that can expand as we see what types of questions people actually have

In addition, I'll take a pass at existing documentation to make it better, including adding a section to the README that links to all of the documentation files as well as have an explanation on setting up a RemoteWorker in CUSTOM_DEPLOYMENT.

Add CI state checks in addition to review state for manager.

This is a great project! My organization will really benefit from something like this. Any chance you can publish documentation exercising other actions we can take such as generating code, library upgrades, and implementing a custom Transformer?

thank you

Make GithubRepo able to handle creating pull requests when one of the repos involved is a fork.

Adding an initialize script command will make getting started a bit easier. Specifically, I want to support a couple things:

• Set up the config.ini file if not present.
• Use a data/custom_components.json file to create a custom components module that can contain all components that are desired.

File format:

[
    {
        "component": "batcher",
        "type": "single",
        "module": "autotransform.batcher.single",
        "class": "SingleBatcher",
        "package": "AutoTransform"
    },
    ...
]

The values above would be replaced with the custom component's values (i.e. input if it's an input component).

The initialize script would also pip install the required packages. We could create a schema index like PyPI that would make it easy for share and distribute schemas/components.

Hey @nathro, interesting project 😉
I maintain libcst so I might be biased but it seems like having a transformer that applies a LibCST codemod would be a great fit.
Let me know if this sounds interesting!

Something broke the migration scripts between when I was testing and when I released them, so I need to fix them.

Add a no_empty_batches param to SingleBatcher so that it can have an option to not create a batch when no items are present.

Add a system for change management for changes triggered by AutoTransform. The way this will work is the following:

A changemanager.py script will be run that leverages a ChangeRequestManager class to do the following:

• Get all existing outstanding change requests (A generic name for a pull request)
• Handle outstanding change requests
  • If the request is more than X days stale, trigger a "rebase" which effectively spawns a worker to run on the same input/schema on a more recent checkout of master, then updates the request to the new changes.
  • If the request has been approved, land it (i.e. merge)
  • If the request has been denied, close it

A GithubChangeRequest and GithubChangeRequestManager class will be included for supporting pull requests on Github.

Handling of outstanding requests will be done using ChangeRequestManagerStep classes. Additional steps can be added by an organization to provide their own handling behavior.

In the future, code may be added that will allow organizations to host a webpage that can display information on outstanding change requests. The security/custom stuff required for this may make this too difficult to do generically, meaning organizations will need to host this content themselves if they desire it.

It would be cool if we can set up autotransform to automatically make suggestions on PRs/provide functionality for setting up schemas to run against PRs.