uc-cdis / cdis-data-client Goto Github PK
View Code? Open in Web Editor NEWdata access client
License: Apache License 2.0
data access client
License: Apache License 2.0
[~/Documents/Notes/gen3/gen3-client$ ./gen3-client configure
Error: required flag(s) "cred" not set
Usage:
gen3-client configure [flags]
Examples:
./gen3-client configure --profile=user1 --creds creds.json
Flags:
--cred string Specify the credential file that you want to use
-h, --help help for configure
Global Flags:
--profile string Specify profile to add or edit with --profile user2 (default "default")
stub out python client sdk and test cases
Would be nice if following a gen3-client upload
command, the client provided a link to the file mapping. This is similar to the link you get when you do a git push origin repo
like this:
remote: Create a pull request for 'feat/CMC' on GitHub by visiting:
remote: https://github.com/uc-cdis/ibdgc-dictionary/pull/new/feat/CMC
The gen3-client could provide output like this (see last two lines):
gen3-client upload --profile=datacommons --upload-path=~/Documents/Notes/files/file.bam
Local history file "/Users/username/.gen3/datacommons.json" has opened
Begin parsing all file paths for "~/Documents/Notes/files/file.bam"
Finish parsing all file paths for "~/Documents/Notes/files/file.bam"
The following file(s) has been founded in path "~/Documents/Notes/files/file.bam" and will be uploaded:
/Users/username/Documents/Notes/files/file.bam
Begin parsing all file paths for "/Users/username/.gen3/config"
Finish parsing all file paths for "/Users/username/.gen3/config"
Uploading data ...
file.bam 29 B / 29 B [========================================================================================================================================================] 100.00% 0s
Successfully uploaded file "/Users/username/Documents/files/file.bam" to GUID dg.0896/18d53086-1ce3-48b0-b685-a6c084a849cb.
Local history data updated in /Users/username/.gen3/datacommons_history.json
To map your files to the appropriate nodes in the data model
Please, visit: https://api-endpoint/submission/files/
cdis-client configure --profile
, similar to awscli configuration, it should prompt to ask for creds.~/.cdis/config
[cdis]
access_key=XXX
secret_key=XXX
gdcapi_endpoint=https://website/
cdis-client list projects --profile test
go configure --profile test --creds creds.json
access_token
using refresh_token, save it in the same profile configUsers like to be able to submit files by type as batch, for example all the bam files under a folder, so they want to do gen3-client upload -filepath /folder/*.bam .
gen3-client update-new profile ${profileName} filepath /folder/*.txt
I get the following error when using the gen3-client in Windows command shell:
In this first example, I purposefully use the wrong profile ('bc') for trouble-shooting purposes:
C:\Users\cgmeyer>gen3-client download --profile bc --guid c2774501-e5c4-4acb-8b4b-23fe803205fb --file=./Documents/file_to_download.json
2018/10/29 12:49:35 Download error: The provided guid at url "https://data.braincommons.org/user/data/download/c2774501-e5c4-4acb-8b4b-23fe803205fb" is not found!
This second example is using a legitimate GUID and profile:
C:\Users\cgmeyer>gen3-client download --profile niaid --guid c2774501-e5c4-4acb-8b4b-23fe803205fb --file=./Documents/file_to_download.json
panic: invalid character '<' looking for beginning of value
goroutine 1 [running]:
github.com/uc-cdis/gen3-client/gen3-client/jwt.DecodeJsonFromString(0xc0422ea000, 0xcc3, 0x7cb9c0, 0xc04225e800, 0x400, 0xc042162240)
/home/travis/gopath/src/github.com/uc-cdis/gen3-client/gen3-client/jwt/utils.go:32 +0x106
github.com/uc-cdis/gen3-client/gen3-client/jwt.(*Functions).ParseFenceURLResponse(0xc042065c98, 0xc042162240, 0xc04200c440, 0x38, 0xc0422c2900, 0x41a)
/home/travis/gopath/src/github.com/uc-cdis/gen3-client/gen3-client/jwt/functions.go:104 +0x253
github.com/uc-cdis/gen3-client/gen3-client/jwt.(*Functions).DoRequestWithSignedHeader(0xc042065c98, 0xc04200e0e0, 0x5, 0x0, 0x0, 0xc04200c440, 0x38, 0xc04200c440, 0x38, 0xc04207c580, ...)
/home/travis/gopath/src/github.com/uc-cdis/gen3-client/gen3-client/jwt/functions.go:147 +0x455
github.com/uc-cdis/gen3-client/gen3-client/g3cmd.init.3.func1(0xc042114780, 0xc0420886e0, 0x0, 0x5)
/home/travis/gopath/src/github.com/uc-cdis/gen3-client/gen3-client/g3cmd/download.go:87 +0x17a
github.com/spf13/cobra.(*Command).execute(0xc042114780, 0xc042088690, 0x5, 0x5, 0xc042114780, 0xc042088690)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:766 +0x2c8
github.com/spf13/cobra.(*Command).ExecuteC(0xac6e60, 0x1, 0x1, 0x87379d)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:852 +0x311
github.com/spf13/cobra.(*Command).Execute(0xac6e60, 0xc042044058, 0x0)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:800 +0x32
github.com/uc-cdis/gen3-client/gen3-client/g3cmd.Execute()
/home/travis/gopath/src/github.com/uc-cdis/gen3-client/gen3-client/g3cmd/root.go:27 +0x34
main.main()
/home/travis/gopath/src/github.com/uc-cdis/cdis-data-client/main.go:8 +0x27
C:\Users\cgmeyer>
Currently, a filename must be specified by the user in download mode with the '--file ' option. It would be nice if the '--file' option could be left off download commands, and when it's not specified, either the filename of the file in object storage would be used, or the 'filename' as specified in the metadata.
Use case:
cdis-data-client data download $uuid
to download an object from S3. This will first call user-api to ask for a presigned-url then download the object with the presigned-urlcdis-data-client data upload $uuid
to upload an object from S3.get Send GET HTTP Request for given URI
post Send POST HTTP Request to the gdcapi
put Send PUT HTTP Request to the gdcapi
Hi,
I am trying to download files from PDC bionimbus (ICGC data) and downloading multiple files in parallel causes erratic stalling of file download after downloading only a few hundred Kb.
This happens consistently and often. This occurs on a computing cluster with no internet connectivity issues.
Second issue: Even downloading files one at a time causes them to stall after a few Gb on average. I can never download large files > 10 Gb.
It would be nice to have a function that would return your data access privileges.
For example:
I just configured a profile "dcf" using credentials downloaded from my Profile page on the DCF commons. I could send this command:
gen3-client auth --profile=dcf
And get this returned upon successfully authenticating:
Programs:
DCF [read, read-storage]
Projects:
DCF-CCLE [create, upload, delete, read, read-storage, write-storage]
ABC-xyz123 [read]
...
Get this returned if you have no access:
You don't currently have access to data from any projects at:
https://nci-crdc-demo.datacommons.io (or whatever you entered for --apiendpoint during configure)
Finally, get this returned if you entered bogus credentials or a bogus API endpoint:
Your profile 'dcf' is not configured correctly. Please, check that you've entered the correct API endpoint for your data commons, and ensure your credentials file has not expired.
This would help trouble-shoot data access issues with users.
Scope inclide;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.