kitware / cumulus Goto Github PK

A REST API for creating and using cloud clusters

License: Apache License 2.0

Python 97.47% CMake 0.69% Shell 1.51% HTML 0.34%

cumulus's Introduction

cumulus

A platform for building HPC workflows.

The goal of the project is to provide a platform for developing HPC workflows. Cumulus enables running workflows on traditional or on-demand clusters. It provides the ability to create compute resources on cloud computing platforms such as AWS, and then to provision MPI clusters on top of those resources using Ansible.

Job management is supported through traditional batch job schedulers. We also provide a workflow engine called TaskFlow to allow a collection of Celery tasks to be run as a workflow.

See documentation for more details.

cumulus's People

Contributors

Stargazers

Watchers

Forkers

mgrauer alesgenova psavery lilinji aronhelser scottwittenburg dealenx xupeiwust

cumulus's Issues

Connection error?

Haven't been able to run anythingt. Keep on getting this error:

[10:33:48.601] INFO: Downloading input for "myjob"
[10:33:48.609] ERROR: Job download/upload error: Traceback (most recent call last):
  File "girderclient.py", line 274, in <module>
    main()
  File "girderclient.py", line 270, in main
    config.token, config.url, config.job, config.dir).run()
  File "girderclient.py", line 212, in run
    r = requests.get(job_url, headers=self._headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

I've vagrant destroy'ed and up/prevision'ed a few times to no avail and I've made sure that the ip in the config.json is reachable from ulmus.

Move playbooks from filesystem into girder

make them accessible to celery worker on separate host

Create custom serializer for celery to deal with ObjectId serialization

http://stackoverflow.com/questions/21631878/celery-is-there-a-way-to-write-custom-json-encoder-decoder

Creating a cluster with scheduler type of 'slurm' fails

Fails with 'Unsupported scheduler'

Look at number of models loads we do for certain endpoints

For several endpoints we are loading the same model multiple times, this could be optimized if it becomes necessary.

Look at replacing StarCluster with ansible playbooks

Clean up filtering out of field returned by REST endpoints

I think we can use Girder built-in support ...

Ensure ansible-refactor meets job tracking needs

make sure provisioning manages private/public keys and job submission/tracking is data is available in cluster configs

Look at LDAP authentication

POST to /clusters endpoint says it requires 'template' where the doc says it's optional

By this:

ClusterParameters {
  config{
    ssh {
      UserNameParameter {
        user (string, optional): The ssh user id
      }
    }
  },
  name (string): The name to give the cluster.,
  template (string, optional): The cluster template to use. (ec2 only),
  type (string): The cluster type, either "ec2" or "trad"
}

Sending only name and type should be ok?

For an EC2 deployment we need to create a VPC and subnet ( they are currently hard coded )

Add AWS profile states to documentation of endpoint

'creating', 'available', 'error'

Develop code path for provisioning launched cluster

Proof of concept provisioning script with dynamic inventory discovery - actual provisioning scripts should be handled through gobig related efforts

Relates to #96

Investigate how to determine maximum number of instance a AWS account can create

See if we can get this from boto, so we can prevent the user trying to create clusters that will fail.

Failure of input download is not reported at task level

Install PyFR on VM image

ec2 key pair is not cleaned up when an AWS profile is deleted

Look at using JSON ref to avoid duplication of task steps

Add support for account string

We need to parameterize the parallel environment

StartCluster create a PE called orte by default. However, this many not exist on traditional clusters so we need to pass in a PE when we submit a job.

Take a look at Deltacloud

This looks worth taking a look at:

https://deltacloud.apache.org/rest-api.html

Add support for local cluster through SSH

Refactor packages and modules into more consistent structure

Add possible cluster status to endpoint doc

'created', 'initializing', 'running', 'terminating', 'terminated', 'error'

Add simple command line interface for quick testing

Investigate persistence storage

We don't want to have to move the data after job has run ...

Visualizer data-dir not always in ${HOME}

This is specially true if jobOutputDir is defined.

We probably need to have the job templated with that information using ${HOME} if "jobOutputDir" is not defined.

Provide finer grain task level status

Current when the a task is in the running state the client has to query the task to work out what jobs it has and then listen for their status update. If we provided some sort of hierarchical state for example 'running.pvw.queued' then this might make things a little easier for clients of the API. Or alternative we could break out the state into two parts. For example have /tasks/<task_id>/status ('created', 'running', 'error', etc ) and /tasks/<task_id>/progress ('pvw.queued', 'pvw.error', etc. ). @TristanWright What do you think of this idea?