nerdalize / nerd Goto Github PK
View Code? Open in Web Editor NEWYour personal nerd that takes care of running jobs on the Nerdalize cloud
Home Page: https://www.nerdalize.com/
License: Apache License 2.0
Your personal nerd that takes care of running jobs on the Nerdalize cloud
Home Page: https://www.nerdalize.com/
License: Apache License 2.0
config
struct that can be mocked.configSnapshot
struct that is returned upon calling config.Read()
, that can be used as a snapshot of the actual config filewhen I do nerd dataset list
, I get the datasets in an apparently random order. I would expect to see the most recently created dataset on top. (docker style).
created
and updated
times in the dataset metadata are now based on the client's clock. We probably want to create a /time endpoint on the server and use this for created
and updated
times.
When a nerd login
process doesn't end (waiting on the oauth flow), and another is started. It should recognize this and give handy feedback.
~ ❯❯❯ nerd login --debug
2017/08/08 10:23:18 outputter.go:143: failed to do oauth login
2017/08/08 10:23:18 outputter.go:147: [DEBUG] Underlying error: listen tcp 127.0.0.1:9876: bind: address already in use
failed to spawn local server
github.com/nerdalize/nerd/command.spawnServer
/home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:142
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2197
failed to do oauth login
github.com/nerdalize/nerd/command.(*Login).DoRun
/home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:65
github.com/nerdalize/nerd/command.(*Login).DoRun-fm
/home/dexter/go/src/github.com/nerdalize/nerd/command/login.go:35
github.com/nerdalize/nerd/command.(*command).Run
/home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:121
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run
/home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154
main.main
/home/dexter/go/src/github.com/nerdalize/nerd/main.go:82
runtime.main
/usr/local/go/src/runtime/proc.go:185
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2197
I find it really confusing that this is start. -- taking this action doesn't actually stat the task, I think 'starting' should be reserved for workloads (and workers).
some of the CLI options configuration doesn't suite itself very well for documentation generation
Right now we only support running public docker images.
We need to think of some way to provide credentials to the nerd run
command to handle private docker repo's.
Command & option synopsis and descriptions should start with a capital letter and (at least command descriptions) should end with a full stop.
This will be handy for debugging.
nerd download
overwrites files that are present on disk by default.
We probably want to prompt the user for action, with the possibility of giving a default answer with a CLI flag.
on windows, the client returns with help text instead of the expected behavior.
In client.nerdapi#doRequest we should check if user's internet connection is down when a request fails.
Now everytime nerd cli is being run, the github api is being hit.
This turns into an error when a user does more than 60 api calls an hour.
Limit the version checker.
nerd workload start -d 5039d75ff5d329459c04 quay.io/nerdalize/xstream-promise
fails
but
nerd workload start quay.io/nerdalize/xstream-promise -d 5039d75ff5d329459c04
works
Show how many bytes (chunks) were already present on cloud storage. This will help the user understand why certain parts of their upload went very quick.
It'd be really nice to be able to run through the complete workflow of uploading data, starting a workload, starting a task and downloading data using a single command (or a single line at least).
So either something like:
nerd run nerdalize/wgetworker ./my-input-data ./my-output-data -- some-command
Or possibly by adding some sort of await command that waits for a specific task to finish in combination with piping:
nerd dataset upload ./my-input-data | nerd workload start nerdalize/wgetworker | nerd task start www.nerdalize.com | nerd task await | nerd dataset download ./my-output-data
That would allow super-straightforward, one-line examples that are easy to copy-paste.
in upload_process.go:62
the error returned doesn't properly propagate.
on the CLI, at 'nerd project list' show which project is active
When I try to authenticate to http://localhost:8002/token?service=nce.nerdalize.com
it responds with a 301 to http://localhost:8002/token/?service=nce.nerdalize.com
. The default go client (http.Client
) properly handles the request, except that it doesn't copy the Authorization header.
The specs (https://golang.org/pkg/net/http/#Client) say that the auth header is only copied if the URLs are of the same (sub)domain. This should however be the case (both localhost:8002
).
Debugging tried so far:
I tried to give Sling a custom client:
cl := &http.Client{
CheckRedirect: func(req *Request, via []*Request) error {
fmt.Printf("%v, %v\n", req, req.URL.Host)
return nil
},
}
The host is localhost:8002
, so I don't see why the header isn't copied.
Things to try:
github.com/pkg/errors is not vendored by glide, requires go get
$: nerd task start cd77e3c05eeb94831851 -- python run.py --tag=DlpjoH7yEj --feedback=http://37.97.208.138:4240 -t 1 -s 5242880
failed to decode unexpected HTTP response (401 Unauthorized)
Failure starting task
Happened on live 14-07-2017 between 15:00 and 15:30
Currently AWS returns sometimes 502 timeouts, or 400 errors.
Without handling these, bash scripting is impossible and needs to handle these errors themselves.
$: nerd task start e2919caeacf96a0d334f -- python run.py --tag=ZxnmLYIJZu --feedback=http://37.97.208.138:4240 -t 10
<: Recieved GitHub error message: API rate limit exceeded for 185.82.186.24. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) (https://developer.github.com/v3/#rate-limiting)
failed to start execution: ThrottlingException: Rate exceeded
status code: 400, request id: 2a14a68d-5a71-11e7-9e60-21373bf4d20e
Failure starting task
Maybe another race condition
The usage of nerd dataset upload
is documented as nerd upload
.
See spec.json:
Line 61 in 4927bd0
Should be nerd dataset upload
, no?
Allows us to react more quickly
We want to version the data management process. Right now this lives in the /command/upload.go
and /command/download.go
files.
We probably want a new package that has an version UploadDataset
and DownloadDataset
function:
nerd/
datamanager/
v1/
func UploadDataset(filePath)
func DownloadDataset(datasetID, filePath)
This way when something changes in the way we store datasets (or do chunking), we can create a new version. This versioning ensures that uploading and downloading a dataset is always consistent.
Check what happens when nerd download
is run with a non-existing dataset ID.
This hampers our ability to add a lot of tasks quickly
When I was trying to copy my session.json to a remote server I kept getting errors like:
2017/06/22 12:48:38 outputter.go:147: [DEBUG] Underlying error: invalid character '\n' in string literal
failed to parse config file
-- this is very hard to debug. Actually base64 encoding it, copy and decoding it made it work. But it's funny.
Make a +1 here if you come accross this error again.
If user forgets to "set project" the urls will not correctly resolve
Unlike what posix defines
Running status returns only limited information for each task, decide what is most usefull for the client and show that in the table.
running go run main.go status -v
doesn't show me the request response
Feedback questions:
When a user has created a workload, (or do anything else) on a workload, it would be nice if the workload would be sticky. It will be a more smooth UX
e.g.
$ nerd workload set asd987f6sa
nerd dataset upload ~/Data
nerd task create -- blah1
nerd task create -- blah2
nerd task list
notice I didn't have to copy-paste the workload ID every time.
When downloading sometimes an error occurred (be lucky and have patience)
~/n/t/lifetheuniverseandeverything (master|✚4…) (nerd-auth)ϵ❯❯❯ time -v nerd dataset download e1d00c6b73b78c01f8af ./output/test2 --debug
Recieved GitHub error message: API rate limit exceeded for 185.82.186.24. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) (https://developer.github.com/v3/#rate-limiting)
2017/06/28 11:54:50 dataset_download.go:81: Downloading dataset with ID 'e1d00c6b73b78c01f8af'
2017/06/28 11:54:50 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/e1d00c6b73b78c01f8af HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP
2017/06/28 11:54:50 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 263
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:50 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: kro9eaB9_sgdWrEeNLJli7UfQca8hV8taT8pQesN4wZPpDorjAyvUw==
X-Amzn-Requestid: d38374c7-5be7-11e7-8276-2524d0cb6728
X-Amzn-Trace-Id: sampled=0;root=1-59537cea-af70adc279df21840203e68a
X-Cache: Miss from cloudfront
{"project_id":"dxlb-default","dataset_id":"e1d00c6b73b78c01f8af","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/e1d00c6b73b78c01f8af","project_root":"dxlb-default","upload_expire":0,"upload_status":"SUCCESS","created_at":1498642404}
2017/06/28 11:54:50 client.go:64: [DEBUG] HTTP Request:
POST /v1/projects/dxlb-default/tokens HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP
{"project_id":"dxlb-default"}
2017/06/28 11:54:51 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 740
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:51 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: fqfuE9CZ4HaN-7f_-U6Vzr293KzTHCtAB3qAH65ATjPB8POgmBu4iQ==
X-Amzn-Requestid: d3b448d5-5be7-11e7-84b2-37f180a0157c
X-Amzn-Trace-Id: sampled=0;root=1-59537cea-380621c4e9597ef8d98f6695
X-Cache: Miss from cloudfront
{"aws_region":"eu-west-1","aws_access_key_id":"ASIAINILN6Q6TXZ7OZTA","aws_expiration":"2017-06-28T21:54:51Z","aws_secret_access_key":"+zMAiSavrmcwYEgUSYm8Mid1DuXFNKbvSCV0wLAp","aws_session_token":"FQoDYXdzEFsaDHWpnbK34pM4bQ84tSL1AqneWwPCsmFNQxJ2dwnliLuABtcemjsWF58B10YfUfKW5jrZOjZ3W323ETs3te4f4xp54nxDmBUqYkBCZfH3R9OFgLqHxmh9Z6+nCL8RBqUqKd4pm8MUkq1EH6CyCVT0KLeXORRoqKjkfGu5SuqLdebhb2owKf3OtIyjcnb4t/xsSkE+dG5P3WGIocj1UHi1nGtNhwV0zCYKDA5dTk2ngdnx+zh7gn1FSDjecyjz37qUvTE2MiJj968w3XSjEwpjDp6pAgM2nzFa6Qg6AwsKiVge5JIdqNVtUZpzdo0f3HNV97adBo/EbtRgGTfGlyg8fc0Qqn8k8YWJnNVsdOmQkb3Wy4Oxh47LYk2rcBOpboeaRW5Uq+U9kj841LrdCj+8ZY1l5ZbbpMt6yCADlt9DJlwIWfRYNfhUTdp3WgJ65X+lHhtfd1H1Oa5aQNU4OB9Ajbsx6dmeUkP09P3n950pRyWcpMkpEE667+flnXsUyNaRELwZoTIo6/nNygU="}
2017/06/28 11:54:51 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/e1d00c6b73b78c01f8af HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzM4NCJ9.eyJpc3MiOiJhdXRoLm5lcmRhbGl6ZS5jb20iLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJsaXN0IiwicnJlY2VpdmUiLCJkZXNjcmliZSIsInN0b3AiLCJzdGFydCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJkb3dubG9hZCIsImNyZWF0ZSIsImxpc3QiLCJ1cGxvYWQiLCJkZXNjcmliZSIsInVwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImRlbGV0ZSIsImxpc3QiLCJ1cGRhdGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODY0MzcyNywibmJmIjoxNDk4NjQzMzY3LCJpYXQiOjE0OTg2NDM0MjcsInN1YiI6IjY4In0.a8dea4LbuZoUnFK3C2Pail3FBkzekgSCHr3Ze82nRkke4h_XkQlGlz4XuDw-BPzHUIrmyVRJNN3HygSpHKCywLqhauVv_I-x3C2S857FoOR80PNFSuoMvKq4td5iw4WP
0 B / 600.00 MiB [-----------------------------------------------------------------------------------------------------------------------] 0.00%2017/06/28 11:54:51 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 263
Content-Type: application/json
Date: Wed, 28 Jun 2017 09:54:51 GMT
Via: 1.1 7f4dc1bba7bed3781d0d492ed99de258.cloudfront.net (CloudFront)
X-Amz-Cf-Id: QlqsrvN99F5vy9zuLeIPhEN-H4Xh9_l50LbBBpFHpuBoyebYivujFQ==
X-Amzn-Requestid: d42ea93c-5be7-11e7-ac2d-4bd068f7c3f2
X-Amzn-Trace-Id: sampled=0;root=1-59537ceb-8fa502c7234d53f09af981bd
X-Cache: Miss from cloudfront
{"project_id":"dxlb-default","dataset_id":"e1d00c6b73b78c01f8af","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/e1d00c6b73b78c01f8af","project_root":"dxlb-default","upload_expire":0,"upload_status":"SUCCESS","created_at":1498642404}
210.12 MiB / 600.00 MiB [=====================================>----------------------------------------------------------------------] 35.02% 39s2017/06/28 12:00:09 outputter.go:143: failed to download dataset 'e1d00c6b73b78c01f8af'
210.12 MiB / 600.00 MiB [=====================================>--------------------------------------------------------------------] 35.02% 5m18s2017/06/28 12:00:09 outputter.go:147: [DEBUG] Underlying error: read tcp 10.1.10.159:39778->52.218.20.36:443: read: connection reset by peer
failed to copy chunk to byte buffer
github.com/nerdalize/nerd/nerd/service/datatransfer/v1.downloadChunks.func1
/home/dexter/go/src/github.com/nerdalize/nerd/nerd/service/datatransfer/v1/download_process.go:91
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2197
failed to download dataset 'e1d00c6b73b78c01f8af'
github.com/nerdalize/nerd/command.(*Download).DoRun
/home/dexter/go/src/github.com/nerdalize/nerd/command/dataset_download.go:101
github.com/nerdalize/nerd/command.(*Download).DoRun-fm
/home/dexter/go/src/github.com/nerdalize/nerd/command/dataset_download.go:35
github.com/nerdalize/nerd/command.(*command).Run
/home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:103
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run
/home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154
main.main
/home/dexter/go/src/github.com/nerdalize/nerd/main.go:75
runtime.main
/usr/local/go/src/runtime/proc.go:185
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2197
Command exited with non-zero status 1
Command being timed: "nerd dataset download e1d00c6b73b78c01f8af ./output/test2 --debug"
User time (seconds): 3.07
System time (seconds): 3.80
Percent of CPU this job got: 2%
Elapsed (wall clock) time (h:mm:ss or m:ss): 5:20.26
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 292664
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 54610
Voluntary context switches: 330951
Involuntary context switches: 2450
Swaps: 0
File system inputs: 263
File system outputs: 258451
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 1
The worker blocks on docker run
. When docker has to pull a remote image, this will take a while. This causes the worker to miss an AWS heartbeat.
We should not wait on docker run
to finish (don't block). We should however check that the container that is pulling is present in the docker ps
list, such that we can send a valid heartbeat message for it.
git clone [email protected]:nerdalize/ne
./make.sh test
result:
=== RUN TestHandleClientError
--- PASS: TestHandleClientError (0.00s)
=== RUN TestHandleError
--- PASS: TestHandleError (0.00s)
PASS
ok github.com/nerdalize/nerd/command 0.013s
? github.com/nerdalize/nerd/nerd [no test files]
? github.com/nerdalize/nerd/nerd/aws [no test files]
=== RUN TestDoRequest
--- PASS: TestDoRequest (0.00s)
PASS
ok github.com/nerdalize/nerd/nerd/client 0.016s
=== RUN TestParseECDSAPublicKeyFromPemBytes
--- FAIL: TestParseECDSAPublicKeyFromPemBytes (0.00s)
nerd_token_test.go:19: Failed to parse valid public key. Error message: failed to parse PEM block containing the public key
=== RUN TestDecodeToken
--- FAIL: TestDecodeToken (0.00s)
nerd_token_test.go:51: expired valid: expected success but got error 'failed to parse public key PEM to ecdsa key: failed to parse PEM block containing the public key'
nerd_token_test.go:66: json parse error: expected error message to contain 'failed to parse nerd token' but error message was 'failed to parse public key PEM to ecdsa key: failed to parse PEM block containing the public key'
FAIL
FAIL github.com/nerdalize/nerd/nerd/client/credentials 0.013s
=== RUN TestEnvProviderRetrieve
--- PASS: TestEnvProviderRetrieve (0.08s)
PASS
ok github.com/nerdalize/nerd/nerd/client/credentials/provider 0.093s
=== RUN TestFromFile
--- PASS: TestFromFile (0.00s)
PASS
ok github.com/nerdalize/nerd/nerd/conf 0.010s
? github.com/nerdalize/nerd/nerd/payload [no test files]
currently it gives feedback as 'page not found'
Lets put some effort in designing the chunk's index header, this will be important for countering long term compatibility issues.
The dataset size (in bytes) is stored in the metadata file. This size is calculated client-side. We should do this server-side to be more reliable.
Right now when logs are written to disk, it writes the same as is outputted to Stdout. So when the -v flag is not set, only very minimal information is logged.
We should log all statements (including Debug) to disk.
Here's the error output
2.50 KiB / 2.50 KiB [=====================================================================================================================] 100.00% 0s
2017/06/23 17:30:46 workload_download.go:103: Downloading dataset with ID 'f0b26998cef7b3c4fc3a'
2017/06/23 17:30:46 client.go:64: [DEBUG] HTTP Request:
GET /v1/projects/dxlb-default/datasets/f0b26998cef7b3c4fc3a HTTP/1.1
Host: batch.nerdalize.com
Authorization: Bearer eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE0OTgyMzE3MjUsImlzcyI6ImF1dGgubmVyZGFsaXplLmNvbSIsIm5iZiI6MTQ5ODIzMTY2NSwic3ViIjoiNjgiLCJhY2Nlc3MiOlt7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ0YXNrIiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJycmVjZWl2ZSIsImxpc3QiLCJkZXNjcmliZSIsInN0YXJ0Iiwic3RvcCIsInJwcm9ncmVzcyJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJkYXRhc2V0IiwiYWRkcmVzcyI6Im5jZS5uZXJkYWxpemUuY29tIiwicmVzb3VyY2VfaWRlbnRpZmllciI6ImR4bGItZGVmYXVsdCIsInJpZ2h0cyI6WyJ1cGxvYWQiLCJsaXN0IiwiZG93bmxvYWQiLCJjcmVhdGUiLCJ1cHJvZ3Jlc3MiLCJkZXNjcmliZSJdfSx7InNlcnZpY2UiOiJuY2UubmVyZGFsaXplLmNvbSIsInR5cGUiOiJ3b3JrZXIiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJ1cGRhdGUiLCJkZWxldGUiXX0seyJzZXJ2aWNlIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJ0eXBlIjoid29ya2xvYWQiLCJhZGRyZXNzIjoibmNlLm5lcmRhbGl6ZS5jb20iLCJyZXNvdXJjZV9pZGVudGlmaWVyIjoiZHhsYi1kZWZhdWx0IiwicmlnaHRzIjpbImNyZWF0ZSIsImxpc3QiLCJkZWxldGUiLCJkZXNjcmliZSJdfV0sImV4cCI6MTQ5ODIzMjAyNX0.RtjOjIRMWSpp_eN_Hp1jvQyYDBjj4KrdyaUojmeXjgxsde4FppyOcKltQpANNFg9THkJif6k4_AtoHO3qC9cudVoDLWVbph2t0-Uqtl7RzqiEs9SQ9-0iN6RBZOCypBW
2017/06/23 17:30:47 client.go:78: [DEBUG] HTTP Response:
HTTP/2.0 200 OK
Content-Length: 272
Content-Type: application/json
Date: Fri, 23 Jun 2017 15:30:47 GMT
Via: 1.1 8e55f0de4d538f549650ba46e729188c.cloudfront.net (CloudFront)
X-Amz-Cf-Id: ck4iusltS8bwOWffMh1-C-n8gu_fIErS2D-XK_ahcAxc3wNML0rAeA==
X-Amzn-Requestid: edcb9e54-5828-11e7-a775-ff4209295690
X-Amzn-Trace-Id: sampled=0;root=1-594d3426-bcfa99649e0cce47fcefb1c5
X-Cache: Miss from cloudfront
{"project_id":"dxlb-default","dataset_id":"f0b26998cef7b3c4fc3a","bucket":"f7c3f734-uniprod-advan.nerdalize.com","dataset_root":"dxlb-default/f0b26998cef7b3c4fc3a","project_root":"dxlb-default","upload_expire":1498231496,"upload_status":"CREATED","created_at":1498231466}
2017/06/23 17:30:47 outputter.go:143: failed to download metadata
2017/06/23 17:30:47 outputter.go:147: [DEBUG] Underlying error: AccessDenied: Access Denied
status code: 403, request id: 4734341EFF4CAEB6, host id: 6npFS5vhuDJrCbj3OgWbEBMJsb+6LMetHGAByfO6iaCMLRyIg2Wt/ewL032rLjmccPtxXnsccPU=
failed to download 'dxlb-default/f0b26998cef7b3c4fc3a/metadata'
failed to download metadata
failed to download metadata
github.com/nerdalize/nerd/nerd/service/datatransfer/v1.GetRemoteDatasetSize
/home/dexter/go/src/github.com/nerdalize/nerd/nerd/service/datatransfer/v1/download.go:69
github.com/nerdalize/nerd/command.(*WorkloadDownload).DoRun
/home/dexter/go/src/github.com/nerdalize/nerd/command/workload_download.go:107
github.com/nerdalize/nerd/command.(*WorkloadDownload).DoRun-fm
/home/dexter/go/src/github.com/nerdalize/nerd/command/workload_download.go:31
github.com/nerdalize/nerd/command.(*command).Run
/home/dexter/go/src/github.com/nerdalize/nerd/command/command.go:103
github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli.(*CLI).Run
/home/dexter/go/src/github.com/nerdalize/nerd/vendor/github.com/mitchellh/cli/cli.go:154
main.main
/home/dexter/go/src/github.com/nerdalize/nerd/main.go:75
runtime.main
/usr/local/go/src/runtime/proc.go:185
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2197
Right now we're checking if a JWT has expired in the CLI. This should happen on the server instead, because we don't want to rely on the user's clock.
This will have implications for the credentials provider design, because it relies on checking if a JWT is expired.
When using the -v flag the flag parser in main.go init()
shows unknown flag 'v'
. We should suppress this output.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.