paultag / moxie

moxie

Moxie

Long-running periodic job scheduler and runner.

moxie's Issues

fix timezone issues

if you're not hosting in UTC, I think it's off a bit.

allow multiple job maintainers

or at least subscriptions?

alerts

add alerts

• email (critical)
• webhooks
• slack
• kgb
• github

bug team docker about log purging

so we don't have to remove the container every time. which isn't a big deal, but it's another moving part.

double check there's no security stuff hanging out

it'd be bad to get pwned.

alternatively, see if websockets tolerates htaccess

params

allow launching a build with a paramater; needed for stuff like openstates historical scrape

implement container.missing.html

ability to consume github webhooks

Possibly just webhooks in general, but GitHub web hooks would be really handy to be able to have jobs fire off in response to pushes, comments, PRs, issues, etc.

setup instructions

(moxie-init/load in a container, pupa init in a container, etc)

order runs in reverse order

clean up postgres work

aiopg is really ugly. make it less ugly

create alert core

for job results and stuff

allow removing old logs

report pull progress / states

since that's currently blind before start - might look like the daemon died when it didn't

complete ssh gateway

expose all of the db over ssh

• show running containers
• enable/disable jobs
• reload yaml somehow (sftp?)
• cli invocation ssh localhost -p 2222 kill test
• ensure \r\n everywhere
• events streaming
• log attach
• notification
• FANCY FUTURE SHIT

container end of log exception for streaming

I remember seeing some view exceptions. handle EOD gracefully. likely related to aiohttp api change

write snazzy tool to do delete of existing jobs

not sure how this'd work just yet, but magic. do we want to remove everything related?

Allow JSON to specify services

so that you can change the impl that's created

run container purges

docker rm docker ps -a | grep Exit | awk '{print $1}'

run image purges

docker rmi docker images | grep "<none>" | awk '{print $3}'

show last 5 scrape statuses on job views

show container status on the ui

online / offline

yaml simplification

ansible is amazing. shared variables is amazing. I hate setting SUNLIGHT_API_KEY 50 times.

concurent start limit

rate limit the number of jobs being brought up at once. Let's do 2 at a time at most?

rename to something a bit easier to remember

find a way to wake the daemon up if there's a db update

reload config if there's a db change or something.

error handling / recovery

communicate exceptions, such as missing images, or unlinkable bits.

aiodocker can't see aiohttp during pip install

During pip install -r requirements.txt, aiohttp installs successfully from git, but then aiodocker doesn't see it during install/compilation:

Obtaining aiohttp from git+https://github.com/KeepSafe/aiohttp#egg=aiohttp (from -r requirements.txt (line 6))
  Updating /home/eric/.virtualenvs/moxie/src/aiohttp clone
  Running setup.py (path:/home/eric/.virtualenvs/moxie/src/aiohttp/setup.py) egg_info for package aiohttp

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching 'aiohttp/_multidict.html'
    warning: no previously-included files found matching 'aiohttp/_multidict.*.so'
    warning: no previously-included files found matching 'aiohttp/_multidict.pyd'
    warning: no previously-included files found matching 'aiohttp/_multidict.*.pyd'
    warning: no previously-included files found matching 'aiohttp/_websocket.html'
    warning: no previously-included files found matching 'aiohttp/_websocket.*.so'
    warning: no previously-included files found matching 'aiohttp/_websocket.pyd'
    warning: no previously-included files found matching 'aiohttp/_websocket.*.pyd'
  Installing extra requirements: 'egg'
Downloading/unpacking aiodocker (from -r requirements.txt (line 7))
  Downloading aiodocker-0.6.tar.gz
  Running setup.py (path:/home/eric/.virtualenvs/moxie/build/aiodocker/setup.py) egg_info for package aiodocker
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/eric/.virtualenvs/moxie/build/aiodocker/setup.py", line 4, in <module>
        from aiodocker import __version__
      File "/home/eric/.virtualenvs/moxie/build/aiodocker/aiodocker/__init__.py", line 3, in <module>
        from .docker import Docker
      File "/home/eric/.virtualenvs/moxie/build/aiodocker/aiodocker/docker.py", line 5, in <module>
        import aiohttp
    ImportError: No module named 'aiohttp'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/eric/.virtualenvs/moxie/build/aiodocker/setup.py", line 4, in <module>

    from aiodocker import __version__

  File "/home/eric/.virtualenvs/moxie/build/aiodocker/aiodocker/__init__.py", line 3, in <module>

    from .docker import Docker

  File "/home/eric/.virtualenvs/moxie/build/aiodocker/aiodocker/docker.py", line 5, in <module>

    import aiohttp

ImportError: No module named 'aiohttp'

write snazzy tool to do update of existing jobs

right now it checks on the pk and skips if it exists

look at use of Session()

I think we're making too many connections to postgres.

I think that might cause 500s when we use too many.

create auth'd api to manage moxie server

re-run jobs, etc

break when image pull failes due to out of disk

better cron?

so that we can set a static time for the next interval

create a logo

allow job 'tagging'

so we can sort by project, etc

ability to "git pull" + "docker build" instead of just "docker pull" for a job

It'd be great to hand a job a Git repo path and have it do git pull and then docker build directly instead of having to wait for either Hub automated builds or pushes from my local box in order to deploy.

fix up log streaming

it broke

yaml fk based on unique key not pk

using the integers was a hack. use a unique pk for each - e.g. email on maintainer

hunt for memory leaks like crazy

just be sure we don't hang coroutines anywhere. I don't think we do, but it's worth checking

make pages not ugly

serve on unix socket

ugh

create r/w API

für a mobile app

multiple web workers

having one moxied is fine, since it's pretty lazy / slow, but letting it handle many workers is super needed.

somehow find a way to let tasks write data back to moxie

perhaps capture a high file descriptor? might need some docker tweaks.

use links

skydns / skydock just took down my server

TIME TRAVEL (2001-01-01 appearing in failure emails)

crank-hy totally failed. The run started at 2001-01-01 00:00:00 and
ended at 2001-01-01 00:00:00.

multiple job servers

The ability to have multiple servers running jobs would be a pretty big improvement, especially if they can all have a single web UI for management.

We talked on IRC, and tagging (ala #29) is probably the cleanest way to get to this.

irc version of the slack bot

https://github.com/devunt/aioirc is a thing, and it might be useful for this (haven't dug into it)

one job against a whole cluster or in many configurations

Running the same job against a whole cluster of machines (or even just with slightly different configuration, like a different GOOS/GOARCH set, for example 😉) would be an interesting way to get a build matrix of sorts.

(just recording this as an issue so we can discuss further ideas/thoughts on it asynchronously)

pull images pre run