nerdalize / nerd

Your personal nerd that takes care of running jobs on the Nerdalize cloud

Nerd - Nerdalize Command Line Interface

Your personal nerd that takes care of running compute jobs on the Nerdalize cloud.

---

Nerdalize is building a different cloud. Instead of constructing huge datacenters, we're distributing our servers over homes. Homeowners use the residual heat for hot showers and to warm their house, and we don't need to build new infrastructure.

In order to make our cloud resources accessible and easy to use, we've developed a CLI that fits your workflow. Whether you’re a researcher, engineer or developer, it allows you to easily run your computations, simulations and analyses on our cloud infrastructure.

Features:

• Moving datasets from you workstation to the cloud and back is included right into the workflow
• Nerd ensures efficient and quick datatransfers through a deduplication algorithm
• Send in thousands of jobs, Nerd makes sure your resources are used as efficiently as possible
• Package your software using industry-standard Docker containers
• Follows basic CLI conventions to provide a scriptable interface your daily dose of automation goodness

Documentation

To start running your compute on the Nerdalize cloud you'll need to set up an account and download the Nerd CLI itself.

• Quickstarts - To quickly get you up and running.
• Ready-to-use Software - We have application-specific guides for Python or FFmpeg for you to get started.
• CLI Reference - For a reference of all available commands

---

Building from Source

If you would like to contribute to the project it is possible to build the Nerd from source:

1. The CLI is written in Go. Make sure you've installed the language SDK as documented here
2. Checkout the repository in your GOPATH:

   git clone [email protected]:nerdalize/nerd.git $GOPATH/src/github.com/nerdalize/nerd
   

3. Go to the checked out repository and build the binary using the included bash script:

   cd $GOPATH/src/github.com/nerdalize/nerd
   ./make.sh build
   

4. The Nerd CLI is now ready to be used in the $GOPATH/bin directory:

   $GOPATH/bin/nerd
   Usage: nerd [--version] [--help] <command> [<args>]
   
   Available commands are:
   ...
   

nerd's Issues

Args before/after option flags need to be both possible

Unlike what posix defines

Fix debug dataset flag overlap

copy-paste of session.json fails

When I was trying to copy my session.json to a remote server I kept getting errors like:

2017/06/22 12:48:38 outputter.go:147: [DEBUG] Underlying error: invalid character '\n' in string literal
failed to parse config file

-- this is very hard to debug. Actually base64 encoding it, copy and decoding it made it work. But it's funny.

Make a +1 here if you come accross this error again.

`nerd task start` should be `nerd task create`

I find it really confusing that this is start. -- taking this action doesn't actually stat the task, I think 'starting' should be reserved for workloads (and workers).

Logging to disk should include Debug logs

Right now when logs are written to disk, it writes the same as is outputted to Stdout. So when the -v flag is not set, only very minimal information is logged.

We should log all statements (including Debug) to disk.

Nerd task start takes 1sec

This hampers our ability to add a lot of tasks quickly

dont' allow API paths to be created when project is empty

If user forgets to "set project" the urls will not correctly resolve

fix generated option overview

some of the CLI options configuration doesn't suite itself very well for documentation generation

• check the weird cmd-encode-base64 option value
• fix occurences with default value being an slice with a single empty string

Task stay on running but appear to have resulted in a dataset anyway

Maybe another race condition

workload download <workload_Id> fails regularly

Here's the error output

 2.50 KiB / 2.50 KiB [=====================================================================================================================] 100.00% 0s
2017/06/23 17:30:46 workload_download.go:103: Downloading dataset with ID 'f0b26998cef7b3c4fc3a'
2017/06/23 17:30:46 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/f0b26998cef7b3c4fc3a HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE0OTgyMzE3MjUsImlzcyI6ImF1dGgubmVyZGFsaXplLmNvbSIsIm5iZiI6MTQ5ODIzMTY2NSwic3ViIjoiNjgiLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJycmVjZWl2ZSIsImxpc3QiLCJkZXNjcmliZSIsInN0YXJ0Iiwic3RvcCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJ1cGxvYWQiLCJsaXN0IiwiZG93bmxvYWQiLCJjcmVhdGUiLCJ1cHJvZ3Jlc3MiLCJkZXNjcmliZSJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJ1cGRhdGUiLCJkZWxldGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODIzMjAyNX0.RtjOjIRMWSpp_eN_Hp1jvQyYDBjj4KrdyaUojmeXjgxsde4FppyOcKltQpANNFg9THkJif6k4_AtoHO3qC9cudVoDLWVbph2t0-Uqtl7RzqiEs9SQ9-0iN6RBZOCypBW


2017/06/23 17:30:47 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 272
Content-Type: application/json
Date: Fri, 23 Jun 2017 15:30:47 GMT
Via: 1.1 8e55f0de4d538f549650ba46e729188c.cloudfront.net (CloudFront)
X-Amz-Cf-Id: ck4iusltS8bwOWffMh1-C-n8gu_fIErS2D-XK_ahcAxc3wNML0rAeA==
X-Amzn-Requestid: edcb9e54-5828-11e7-a775-ff4209295690
X-Amzn-Trace-Id: sampled=0;root=1-594d3426-bcfa99649e0cce47fcefb1c5
X-Cache: Miss from cloudfront

{"project_id":"dxlb-default","dataset_id":"f0b26998cef7b3c4fc3a","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/f0b26998cef7b3c4fc3a","project_root":"dxlb-default","upload_expire":1498231496,"upload_status":"CREATED","created_at":1498231466}

2017/06/23 17:30:47 outputter.go:143: failed to download metadata
2017/06/23 17:30:47 outputter.go:147: [DEBUG] Underlying error: AccessDenied: Access Denied
        status code: 403, request id: 4734341EFF4CAEB6, host id: 6npFS5vhuDJrCbj3OgWbEBMJsb+6LMetHGAByfO6iaCMLRyIg2Wt/ewL032rLjmccPtxXnsccPU=
failed to download 'dxlb-default/f0b26998cef7b3c4fc3a/metadata'
failed to download metadata
failed to download metadata
github.com/nerdalize/nerd/nerd/service/datatransfer/v1.GetRemoteDatasetSize
        /home/dexter/go/src/github.com/nerdalize/nerd/nerd/service/datatransfer/v1/download.go:69
github.com/nerdalize/nerd/command.(*WorkloadDownload).DoRun
        /home/dexter/go/src/github.com/nerdalize/nerd/command/workload_download.go:107
github.com/nerdalize/nerd/command.(*WorkloadDownload).DoRun-fm
        /home/dexter/go/src/github.com/nerdalize/nerd/command/workload_download.go:31
github.com/nerdalize/nerd/command.(*command).Run
        /home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:103
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run
        /home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154
main.main
        /home/dexter/go/src/github.com/nerdalize/nerd/main.go:75
runtime.main
        /usr/local/go/src/runtime/proc.go:185
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2197

Rate limiting on github is easily triggered (by version check)

Now everytime nerd cli is being run, the github api is being hit.
This turns into an error when a user does more than 60 api calls an hour.

Limit the version checker.

`nerd login` doesn't recognize another nerd login process.

When a nerd login process doesn't end (waiting on the oauth flow), and another is started. It should recognize this and give handy feedback.

~ ❯❯❯ nerd login --debug
2017/08/08 10:23:18 outputter.go:143: failed to do oauth login                                                                       
2017/08/08 10:23:18 outputter.go:147: [DEBUG] Underlying error: listen tcp 127.0.0.1:9876: bind: address already in use              
failed to spawn local server                                                                                                         
github.com/nerdalize/nerd/command.spawnServer                                                                                        
        /home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:142                                                           
runtime.goexit                                                                                                                       
        /usr/local/go/src/runtime/asm_amd64.s:2197                                                                                   
failed to do oauth login                                                                                                             
github.com/nerdalize/nerd/command.(*Login).DoRun                                                                                     
        /home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:65                                                            
github.com/nerdalize/nerd/command.(*Login).DoRun-fm                                                                                  
        /home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:35                                                            
github.com/nerdalize/nerd/command.(*command).Run                                                                                     
        /home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:121                                                         
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run                                                                 
        /home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154                                     
main.main                                                                                                                            
        /home/dexter/go/src/github.com/nerdalize/nerd/main.go:82                                                                     
runtime.main                                                                                                                         
        /usr/local/go/src/runtime/proc.go:185                                                                                        
runtime.goexit                                                                                                                       
        /usr/local/go/src/runtime/asm_amd64.s:2197  

Make workload 'sticky'

When a user has created a workload, (or do anything else) on a workload, it would be nice if the workload would be sticky. It will be a more smooth UX

e.g.

$ nerd workload set asd987f6sa

nerd dataset upload ~/Data
nerd task create -- blah1
nerd task create -- blah2

nerd task list

notice I didn't have to copy-paste the workload ID every time.

Refactor conf package

• Create a config struct that can be mocked.
• Create a configSnapshot struct that is returned upon calling config.Read(), that can be used as a snapshot of the actual config file
• Only write fields that diff from the defaults

Hide flag parsing output in init()

When using the -v flag the flag parser in main.go init() shows unknown flag 'v'. We should suppress this output.

Redirects are not properly handled (Authorization header is not set)

When I try to authenticate to http://localhost:8002/token?service=nce.nerdalize.com it responds with a 301 to http://localhost:8002/token/?service=nce.nerdalize.com. The default go client (http.Client) properly handles the request, except that it doesn't copy the Authorization header.
The specs (https://golang.org/pkg/net/http/#Client) say that the auth header is only copied if the URLs are of the same (sub)domain. This should however be the case (both localhost:8002).

Debugging tried so far:
I tried to give Sling a custom client:

cl := &http.Client{
  CheckRedirect: func(req *Request, via []*Request) error {
    fmt.Printf("%v, %v\n", req, req.URL.Host)
    return nil
  },
}

The host is localhost:8002, so I don't see why the header isn't copied.

Things to try:

• Check if header is really set
• Use a debugger to step trough go native code

show which project is active

on the CLI, at 'nerd project list' show which project is active

Check ./make.sh test upon clear checkout

git clone [email protected]:nerdalize/ne
./make.sh test

result:

=== RUN   TestHandleClientError
--- PASS: TestHandleClientError (0.00s)
=== RUN   TestHandleError
--- PASS: TestHandleError (0.00s)
PASS
ok  	github.com/nerdalize/nerd/command	0.013s
?   	github.com/nerdalize/nerd/nerd	[no test files]
?   	github.com/nerdalize/nerd/nerd/aws	[no test files]
=== RUN   TestDoRequest
--- PASS: TestDoRequest (0.00s)
PASS
ok  	github.com/nerdalize/nerd/nerd/client	0.016s
=== RUN   TestParseECDSAPublicKeyFromPemBytes
--- FAIL: TestParseECDSAPublicKeyFromPemBytes (0.00s)
	nerd_token_test.go:19: Failed to parse valid public key. Error message: failed to parse PEM block containing the public key
=== RUN   TestDecodeToken
--- FAIL: TestDecodeToken (0.00s)
	nerd_token_test.go:51: expired valid: expected success but got error 'failed to parse public key PEM to ecdsa key: failed to parse PEM block containing the public key'
	nerd_token_test.go:66: json parse error: expected error message to contain 'failed to parse nerd token' but error message was 'failed to parse public key PEM to ecdsa key: failed to parse PEM block containing the public key'
FAIL
FAIL	github.com/nerdalize/nerd/nerd/client/credentials	0.013s
=== RUN   TestEnvProviderRetrieve
--- PASS: TestEnvProviderRetrieve (0.08s)
PASS
ok  	github.com/nerdalize/nerd/nerd/client/credentials/provider	0.093s
=== RUN   TestFromFile
--- PASS: TestFromFile (0.00s)
PASS
ok  	github.com/nerdalize/nerd/nerd/conf	0.010s
?   	github.com/nerdalize/nerd/nerd/payload	[no test files]

task list output should show shortened command

When I add a sequence as task input, the entire sequence shows. It doesn't look pretty :)

`nerd dataset download` sometimes hangs

When downloading sometimes an error occurred (be lucky and have patience)

~/n/t/lifetheuniverseandeverything (master|✚4…) (nerd-auth)ϵ❯❯❯ time -v nerd dataset download e1d00c6b73b78c01f8af ./output/test2 --debug
Recieved GitHub error message: API rate limit exceeded for 185.82.186.24. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) (https://developer.github.com/v3/#rate-limiting)
2017/06/28 11:54:50 dataset_download.go:81: Downloading dataset with ID 'e1d00c6b73b78c01f8af'
2017/06/28 11:54:50 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/e1d00c6b73b78c01f8af HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP


2017/06/28 11:54:50 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 263
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:50 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: kro9eaB9_sgdWrEeNLJli7UfQca8hV8taT8pQesN4wZPpDorjAyvUw==
X-Amzn-Requestid: d38374c7-5be7-11e7-8276-2524d0cb6728
X-Amzn-Trace-Id: sampled=0;root=1-59537cea-af70adc279df21840203e68a
X-Cache: Miss from cloudfront

{"project_id":"dxlb-default","dataset_id":"e1d00c6b73b78c01f8af","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/e1d00c6b73b78c01f8af","project_root":"dxlb-default","upload_expire":0,"upload_status":"SUCCESS","created_at":1498642404}

2017/06/28 11:54:50 client.go:64: [DEBUG] HTTP Request:
POST /v1/projects/dxlb-default/tokens HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP

{"project_id":"dxlb-default"}

2017/06/28 11:54:51 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 740
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:51 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: fqfuE9CZ4HaN-7f_-U6Vzr293KzTHCtAB3qAH65ATjPB8POgmBu4iQ==
X-Amzn-Requestid: d3b448d5-5be7-11e7-84b2-37f180a0157c
X-Amzn-Trace-Id: sampled=0;root=1-59537cea-380621c4e9597ef8d98f6695
X-Cache: Miss from cloudfront

{"aws_region":"eu-west-1","aws_access_key_id":"ASIAINILN6Q6TXZ7OZTA","aws_expiration":"2017-06-28T21:54:51Z","aws_secret_access_key":"+zMAiSavrmcwYEgUSYm8Mid1DuXFNKbvSCV0wLAp","aws_session_token":"FQoDYXdzEFsaDHWpnbK34pM4bQ84tSL1AqneWwPCsmFNQxJ2dwnliLuABtcemjsWF58B10YfUfKW5jrZOjZ3W323ETs3te4f4xp54nxDmBUqYkBCZfH3R9OFgLqHxmh9Z6+nCL8RBqUqKd4pm8MUkq1EH6CyCVT0KLeXORRoqKjkfGu5SuqLdebhb2owKf3OtIyjcnb4t/xsSkE+dG5P3WGIocj1UHi1nGtNhwV0zCYKDA5dTk2ngdnx+zh7gn1FSDjecyjz37qUvTE2MiJj968w3XSjEwpjDp6pAgM2nzFa6Qg6AwsKiVge5JIdqNVtUZpzdo0f3HNV97adBo/EbtRgGTfGlyg8fc0Qqn8k8YWJnNVsdOmQkb3Wy4Oxh47LYk2rcBOpboeaRW5Uq+U9kj841LrdCj+8ZY1l5ZbbpMt6yCADlt9DJlwIWfRYNfhUTdp3WgJ65X+lHhtfd1H1Oa5aQNU4OB9Ajbsx6dmeUkP09P3n950pRyWcpMkpEE667+flnXsUyNaRELwZoTIo6/nNygU="}

2017/06/28 11:54:51 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/e1d00c6b73b78c01f8af HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP


 0 B / 600.00 MiB [-----------------------------------------------------------------------------------------------------------------------]   0.00%2017/06/28 11:54:51 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 263
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:51 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: QlqsrvN99F5vy9zuLeIPhEN-H4Xh9_l50LbBBpFHpuBoyebYivujFQ==
X-Amzn-Requestid: d42ea93c-5be7-11e7-ac2d-4bd068f7c3f2
X-Amzn-Trace-Id: sampled=0;root=1-59537ceb-8fa502c7234d53f09af981bd
X-Cache: Miss from cloudfront

{"project_id":"dxlb-default","dataset_id":"e1d00c6b73b78c01f8af","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/e1d00c6b73b78c01f8af","project_root":"dxlb-default","upload_expire":0,"upload_status":"SUCCESS","created_at":1498642404}

 210.12 MiB / 600.00 MiB [=====================================>----------------------------------------------------------------------]  35.02% 39s2017/06/28 12:00:09 outputter.go:143: failed to download dataset 'e1d00c6b73b78c01f8af'
 210.12 MiB / 600.00 MiB [=====================================>--------------------------------------------------------------------]  35.02% 5m18s2017/06/28 12:00:09 outputter.go:147: [DEBUG] Underlying error: read tcp 10.1.10.159:39778->52.218.20.36:443: read: connection reset by peer
failed to copy chunk to byte buffer
github.com/nerdalize/nerd/nerd/service/datatransfer/v1.downloadChunks.func1
        /home/dexter/go/src/github.com/nerdalize/nerd/nerd/service/datatransfer/v1/download_process.go:91
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2197
failed to download dataset 'e1d00c6b73b78c01f8af'
github.com/nerdalize/nerd/command.(*Download).DoRun
        /home/dexter/go/src/github.com/nerdalize/nerd/command/dataset_download.go:101
github.com/nerdalize/nerd/command.(*Download).DoRun-fm
        /home/dexter/go/src/github.com/nerdalize/nerd/command/dataset_download.go:35
github.com/nerdalize/nerd/command.(*command).Run
        /home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:103
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run
        /home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154
main.main
        /home/dexter/go/src/github.com/nerdalize/nerd/main.go:75
runtime.main
        /usr/local/go/src/runtime/proc.go:185
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2197
Command exited with non-zero status 1
        Command being timed: "nerd dataset download e1d00c6b73b78c01f8af ./output/test2 --debug"
        User time (seconds): 3.07
        System time (seconds): 3.80
        Percent of CPU this job got: 2%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 5:20.26
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 292664
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 1
        Minor (reclaiming a frame) page faults: 54610
        Voluntary context switches: 330951
        Involuntary context switches: 2450
        Swaps: 0
        File system inputs: 263
        File system outputs: 258451
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 1

Nerd download overwrite files

nerd download overwrites files that are present on disk by default.
We probably want to prompt the user for action, with the possibility of giving a default answer with a CLI flag.

Allow running private docker images

Right now we only support running public docker images.
We need to think of some way to provide credentials to the nerd run command to handle private docker repo's.

How to handle non-existing input datasets

Check what happens when nerd download is run with a non-existing dataset ID.

dataset upload doesn't work (windows)

on windows, the client returns with help text instead of the expected behavior.

Make api handle network errors

Currently AWS returns sometimes 502 timeouts, or 400 errors.
Without handling these, bash scripting is impossible and needs to handle these errors themselves.

$: nerd task start e2919caeacf96a0d334f -- python run.py --tag=ZxnmLYIJZu --feedback=http://37.97.208.138:4240 -t 10
<: Recieved GitHub error message: API rate limit exceeded for 185.82.186.24. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) (https://developer.github.com/v3/#rate-limiting)
failed to start execution: ThrottlingException: Rate exceeded
        status code: 400, request id: 2a14a68d-5a71-11e7-9e60-21373bf4d20e
Failure starting task

Enable Alerting of Lambda Errors

Allows us to react more quickly

With upload show the amount of dedup bytes

Show how many bytes (chunks) were already present on cloud storage. This will help the user understand why certain parts of their upload went very quick.

Povide user feedback when internet connection is down.

In client.nerdapi#doRequest we should check if user's internet connection is down when a request fails.

decide on what information is stored in for the status command

Running status returns only limited information for each task, decide what is most usefull for the client and show that in the table.

better feedback when project is not set

currently it gives feedback as 'page not found'

-v doesn't work on status command

running go run main.go status -v doesn't show me the request response

Settle on a plan for providing feedback to the user for commands

Feedback questions:

• progress?
• show files (how many)?
• show total size?
• add a debug flag
• more?
• make sure cmd.ui isnt nil

Error in upload_process.go doesn't propagate to fail command

in upload_process.go:62 the error returned doesn't properly propagate.

JWT expiration not client side

Right now we're checking if a JWT has expired in the CLI. This should happen on the server instead, because we don't want to rely on the user's clock.

This will have implications for the credentials provider design, because it relies on checking if a JWT is expired.

Versioned datamanager package

We want to version the data management process. Right now this lives in the /command/upload.go and /command/download.go files.

We probably want a new package that has an version UploadDataset and DownloadDataset function:

nerd/
  datamanager/
    v1/
      func UploadDataset(filePath)
      func DownloadDataset(datasetID, filePath)

This way when something changes in the way we store datasets (or do chunking), we can create a new version. This versioning ensures that uploading and downloading a dataset is always consistent.

Package not vendored

github.com/pkg/errors is not vendored by glide, requires go get

Create output directory for nerd download if not exists

Dataset metadata time not client time

created and updated times in the dataset metadata are now based on the client's clock. We probably want to create a /time endpoint on the server and use this for created and updated times.

Provide a simple & quick way to run through the complete workflow

It'd be really nice to be able to run through the complete workflow of uploading data, starting a workload, starting a task and downloading data using a single command (or a single line at least).

So either something like:

nerd run nerdalize/wgetworker ./my-input-data ./my-output-data -- some-command

Or possibly by adding some sort of await command that waits for a specific task to finish in combination with piping:

nerd dataset upload ./my-input-data | nerd workload start nerdalize/wgetworker | nerd task start www.nerdalize.com | nerd task await | nerd dataset download ./my-output-data

That would allow super-straightforward, one-line examples that are easy to copy-paste.

Make sure config.json has read-only permission for current user

Description formatting should be improved

Command & option synopsis and descriptions should start with a capital letter and (at least command descriptions) should end with a full stop.

Worker misses heartbeat because docker run lasts a while

The worker blocks on docker run. When docker has to pull a remote image, this will take a while. This causes the worker to miss an AWS heartbeat.

We should not wait on docker run to finish (don't block). We should however check that the container that is pulling is present in the docker ps list, such that we can send a valid heartbeat message for it.

Usage of dataset upload command is wrongly documented in spec

The usage of nerd dataset upload is documented as nerd upload.
See spec.json:

nerd/spec.json

Line 61 in 4927bd0

"usage": "nerd upload \u003cpath\u003e",

Should be nerd dataset upload, no?

Chunks Index Header information Design

Lets put some effort in designing the chunk's index header, this will be important for countering long term compatibility issues.

• Do we want to remove the times?
• Do we want to add other informaition?

workload start doesn't recognize -d in wrong position

nerd workload start -d 5039d75ff5d329459c04 quay.io/nerdalize/xstream-promise

fails

but
nerd workload start quay.io/nerdalize/xstream-promise -d 5039d75ff5d329459c04
works

Dataset metadata dataset size

The dataset size (in bytes) is stored in the metadata file. This size is calculated client-side. We should do this server-side to be more reliable.

List datasets by time created

when I do nerd dataset list, I get the datasets in an apparently random order. I would expect to see the most recently created dataset on top. (docker style).

Don't show project ID in project list to confuse users

Weird unauthorized response on task create

$: nerd task start cd77e3c05eeb94831851 -- python run.py --tag=DlpjoH7yEj --feedback=http://37.97.208.138:4240 -t 1 -s 5242880
failed to decode unexpected HTTP response (401 Unauthorized)
Failure starting task

Happened on live 14-07-2017 between 15:00 and 15:30

Show in task list on which worker a given task was executed

This will be handy for debugging.