ndar / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.

NDAR

Code for working with NDAR data.

nda-tools's Issues

`-s3` is useless

Despite supposedly being the "preferred" method of downloading data, -s3 is utterly useless because the downloaded files are owned by NDA. This blocks the bucket owner (user) from doing almost anything with the files.

The solution is applying --acl bucket-owner-full-control during the s3-to-s3 copy.

See https://stackoverflow.com/questions/34055084/s3-user-cannot-access-object-in-his-own-s3-bucket-if-created-by-another-user

If my file upload was disrupted, can I cancel it, or delete the process?

Are there any other options for

vtcmd -r

?

I don't want to RESUME my submission; I want to CANCEL it. I already completed it under a separate submissionID.

Question: clarify incremental submissions procedure for neuroimaging data

AFAIK (myself and @amanelis17) neuroimaging data requires incremental uploads. Looking forward to the next submission we wondered how we will need to proceed in the future when more subjects/sessions get collected.
It would be great if README.md made a clear example on how to proceed in those cases. Ideally, if provided image03.csv file was analyzed (based on NDA server side data) for older (already uploaded) entries and only new entries were uploaded, it would be great.

Thank you in advance!

store log files in specific directory

Hello, I have been working on downloading a large series of subjects through an HPC. The log files are being stored within the NDA file located in my home directory however, I only have a limited amount of storage within my home dir and was wondering if there is a specific parameter that could be added that offers users the ability to store the log files in a different location?

Validation does not work without -b flag

For this line to act i.e. the vtcmd.py script to read my password from keyring:

nda-tools/NDATools/clientscripts/vtcmd.py

Line 123 in 6e8255f

config.read_user_credentials(auth_req)

it appears that I need to use the -b flag:

nda-tools/NDATools/clientscripts/vtcmd.py

Lines 46 to 47 in 6e8255f

	parser.add_argument('-b', '--buildPackage', action='store_true',
	help='Flag whether to construct the submission package')

Is this your expected design? I thought validation should work without any password let alone -b flag.

KeyError when parsing server response upon validation error

I received the following KeyError, after attempting to submit.

Traceback (most recent call last):
  File "/home/jacoblee/miniconda3/envs/ndar/bin/vtcmd", line 8, in <module>
    sys.exit(main())
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 284, in main
    validation_results = validate_files(args.files, w, bp, threads=args.workerThreads, config=config)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 181, in validate_files
    validation.output()
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Validation.py", line 136, in output
    column = v['columnName']
KeyError: 'columnName'

The key, columnName is hard-coded in the following bit of code:

               file_name = self.uuid_dict[response['id']]['file']
                if response['errors'] == {}:
                    writer.writerow(
                        {'FILE': file_name, 'ID': response['id'], 'STATUS': response['status'],
                         'EXPIRATION_DATE': response['expiration_date'], 'ERRORS': 'None', 'COLUMN': 'None',
                         'MESSAGE':'None','RECORD': 'None'})
                else:
                    for error, value in response['errors'].items():
                        for v in value:
                            column = v['columnName']
                            message = v['message']
                            try:
                                record = v['recordNumber']
                            except KeyError:
                                record = ' '
                            writer.writerow(
                                {'FILE': file_name, 'ID': response['id'], 'STATUS': response['status'],
                                 'EXPIRATION_DATE': response['expiration_date'], 'ERRORS': error, 'COLUMN': column,
                                 'MESSAGE': message, 'RECORD': record})
            csvfile.close()

It looks like response['errors'] dict did not have the expected keys.

I modified the code to print out the error and error messages:

for error, value in response['errors'].items():
    print(f'{error}: {value}')

and it printed:

data_structure: [{'message': 'File not recognized as a data file, and was not associated or referenced from a data file.'}]

The cause of the error is probably something on my own end, but it does look like the code is not correctly anticipating the structure of the response.

Parallel Submission?

Hello!

Sorry to bother you again. I was just wondering if you know whether it would be possible to run nda-tools in parallel across several processors and then submit the resulting files in parallel to NDA? I'm trying to decrease the upload time, but I'm not sure if doing it in parallel would be a good idea.

Thanks,

John

using `downloadcmd` doesn't work

Taking a hint from https://github.com/ABCD-ReproNim/exercises/blob/main/project_month_prep/downloadMRI.md , I have tried two commands below to download a data package in NIHM, but have failed with different errors.

Attempt 1 :

tried : downloadcmd -dp XX -u XXX -p XXX -wt 8 -v (-dp, -u, -p were changed with XXX for security reasons)

got :

Running NDATools Version 0.2.18
usage: downloadcmd <S3_path_list>
downloadcmd: error: unrecognized arguments: -wt 8 -v

Attempt 2 :

tried : downloadcmd -dp XXX
got :

Running NDATools Version 0.2.18
-u/--username argument not provided. Using default value of 'XXX' which was saved in /home/connectome/dyhan316/.NDATools/settings.cfg
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.local/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 187, in main
    config = configure(args)
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 177, in configure
    config.read_user_credentials()
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/Configuration.py", line 164, in read_user_credentials
    self.password = keyring.get_password(self.SERVICE_NAME, self.username)
  File "/usr/lib/python3/dist-packages/keyring/core.py", line 57, in get_password
    return _keyring_backend.get_password(service_name, username)
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 64, in get_password
    collection = self.get_preferred_collection()
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 58, in get_preferred_collection
    raise KeyringLocked("Failed to unlock the collection!")
keyring.errors.KeyringLocked: Failed to unlock the collection!

I tried to resolve the keyring issue by looking at : https://pypi.org/project/keyring/ and using keyring.set_password('nda-tools', 'YOUR_USERNAME', 'NEW_PASSWORD') command, but got

>>> import keyring
>>> keyring.set_password('nda-tools', 'XXX', 'XXX')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/keyring/core.py", line 63, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 77, in set_password
    collection = self.get_preferred_collection()
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 58, in get_preferred_collection
    raise KeyringLocked("Failed to unlock the collection!")
keyring.errors.KeyringLocked: Failed to unlock the collection!
>>> 

(the usernames and password was replaced with XXX for security reasons)

Additional Information

I realized that when I tired downloading ada-tools, I got the following ERROR:

(download_ABCD) dyhan316@node1:/storage/bigdata/CHA_bigdata/ABCD_release4$ pip install nda-tools
Collecting nda-tools
  Using cached https://files.pythonhosted.org/packages/fc/fb/f4261be968916758294db6336afac6d94b36cca5121c42ce2da3acefe8d3/nda_tools-0.2.18-py3-none-any.whl
Collecting botocore (from nda-tools)
  Using cached https://files.pythonhosted.org/packages/f1/3c/7bf0f6fabc9cb05758b10fe8c688d3f0540bb899abd1cdc4072242ee0b73/botocore-1.27.55-py3-none-any.whl
Requirement already satisfied: packaging in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (19.2)
Requirement already satisfied: requests in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (2.22.0)
Requirement already satisfied: tqdm in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (4.36.1)
Requirement already satisfied: pyyaml in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (5.1.2)
Collecting boto3 (from nda-tools)
  Using cached https://files.pythonhosted.org/packages/9e/78/ecc8c65153a427ff952b868af287f0c9d7f4b269edd21316f05e6ce194b0/boto3-1.24.55-py3-none-any.whl
Requirement already satisfied: keyring in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (18.0.0)
Requirement already satisfied: mock in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (3.0.5)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/anaconda3/lib/python3.7/site-packages (from botocore->nda-tools) (2.8.0)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore->nda-tools)
  Using cached https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl
Collecting urllib3<1.27,>=1.25.4 (from botocore->nda-tools)
  Using cached https://files.pythonhosted.org/packages/d1/cb/4783c8f1a90f89e260dbf72ebbcf25931f3a28f8f80e2e90f8a589941b19/urllib3-1.26.11-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/anaconda3/lib/python3.7/site-packages (from packaging->nda-tools) (1.12.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/anaconda3/lib/python3.7/site-packages (from packaging->nda-tools) (2.4.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (3.0.4)
Collecting s3transfer<0.7.0,>=0.6.0 (from boto3->nda-tools)
  Using cached https://files.pythonhosted.org/packages/5e/c6/af903b5fab3f9b5b1e883f49a770066314c6dcceb589cf938d48c89556c1/s3transfer-0.6.0-py3-none-any.whl
Requirement already satisfied: entrypoints in /usr/anaconda3/lib/python3.7/site-packages (from keyring->nda-tools) (0.3)
Requirement already satisfied: secretstorage; sys_platform == "linux" and python_version >= "3.5" in /usr/anaconda3/lib/python3.7/site-packages (from keyring->nda-tools) (3.1.1)
Requirement already satisfied: cryptography in /usr/anaconda3/lib/python3.7/site-packages (from secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (2.7)
Requirement already satisfied: jeepney in /usr/anaconda3/lib/python3.7/site-packages (from secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (0.4.1)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/anaconda3/lib/python3.7/site-packages (from cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (1.0.1)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/anaconda3/lib/python3.7/site-packages (from cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (1.12.3)
Requirement already satisfied: pycparser in /usr/anaconda3/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (2.19)
ERROR: requests 2.22.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 1.26.11 which is incompatible.
Installing collected packages: jmespath, urllib3, botocore, s3transfer, boto3, nda-tools
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/anaconda3/lib/python3.7/site-packages/jmespath-1.0.1.dist-info'
Consider using the `--user` option or check the permissions.

Could this be the reason for the aforementioned errors? Should I ask the server manager to install nda-tools using root user privileges?

Could anyone please lend a help? Thank you!

Submission of data with custom scope fails with "Unrecognized column name"

Submission of instruments with custom scope fails using the tool, but does not fail when using the online tool. The error log indicates that the columns of the submitted CSV were not recognized as belonging to the data dictionary, e.g.

54fc58-742c-4066-86e9-0eaf0d151ff2,CompleteWithErrors,2021-07-07T11:13:24.900-0400,unrecognizedColumnName,scid_5_pd1,Data column names must all match with the structure element_names or aliases. Unrecognized column names are not permitted in your data file.,

When submitting data with custom scope using nda tools, you enter in the optional --scope argument, which documentation indicates has type string. In particular, it doesn't indicate that it is an integer or a sequence like a list or tuple.

I ran with pdbdebugger and discovered that in the process of parsing the arguments to the validation tool, the scope argument is treated as a list or other subscriptable data structure. In particular, my scope (which is the collection id) went from "5XXXX" to "5", which is possible because in Python strings are subscriptable. To confirm that this was indeed the problem, I modified my local copy to not subscript the scope argument, and I was able to submit the data structure successfully.

504 Server Error: Gateway Time-out when downloading data from large packages

The ABCC dataset (collection 3165) is 168.57TB. If I try to download the entire package with the command downloadcmd -dp ${package_id} it will begin downloading, but then cut off after about a minute and return the following error:

Traceback (most recent call last):
  File "/home/faird/perr0372/.local/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 199, in main
    s3Download.start()
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 327, in start
    for package_file_id_list in self.generate_download_batch_file_ids():
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 595, in generate_download_batch_file_ids
    files = self.get_package_files_by_page(page, batch_size)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 971, in get_package_files_by_page
    raise e
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 961, in get_package_files_by_page
    tmp = get_request(url, headers=self.request_header(), auth=self.auth, error_handler=HttpErrorHandlingStrategy.reraise_status)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Utils.py", line 297, in get_request
    error_handler(tmp)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Utils.py", line 74, in reraise_status
    response.raise_for_status()
  File "/panfs/roc/msisoft/anaconda/python3-2020.07-mamba/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://nda.nih.gov/api/package/1203969/files?page=2&size=50

I don't actually want to download the entire dataset at once, but rather I'd like to be able to download a single subject or a single filetype from all subjects using the --file-regex parameter. When I do this I get the same error as above.

I've tested this out on a much smaller package of around 80GB (essentially just a single submission from the collection) and it works just fine so I'm assuming this error is likely caused by the size of the data package that I am trying to download from.

Any ideas on how I can get around this issue so that users can easily download the specific data that they want from this collection?

Validating files frozen, no errors or updates

I'm submitting eeg_sub_files01 data with manifest files. This is the command I am calling.
vtcmd eeg_sub_files01.txt -m ./manifest -l ./BIDS -t {title} -d {description} -c {COLLECTION_ID} -b

This is the output. Nothing seems to happen ever and I don't get any error messages.

Validating files...
  0%|                                                                                        | 0/1 [00:00<?, ?it/s]

ABCD MRI data downloading issue

Hi experts,

I try to download ABCD MRI data using NDA-tools downloadcmd script.
Following https://github.com/ABCD-ReproNim/exercises/blob/main/project_month_prep/downloadMRI.md All steps above were successful. But it comes to this downloading step:

! mkdir /home/jovyan/ABCDndar/tar_files
! downloadcmd -dp -d '/home/jovyan/ABCDndar/tar_files' -u -p -t '/home/jovyan/ABCDndar/s3_derv_links_5subj.txt'

Now s3_derv_links_5subj.txt contains five s3 links.

1. What is here? how to get it?
2. Are AWS Credentials required? how to get it?

Cheers~
Thank you!

File Fails to Pass Validation Because of GUIDs

Hello all,

Sorry to bother you with this, but I am encountering and issue where my files fail to pass validation because we ended up having to replace some old GUIDs with new GUIDs. This is because we discovered that some of the information (such as middle name and last name) were incorrect for some subjects. We alerted the NDA Helpdesk to this issue and they said that they would replace the old GUIDs with the new, but we still cannot upload the files because of this problem. Is there any workaround?

Thanks!

John

incorrectly handles error messages without columnName field

gory details:

$ vtcmd /data/NDA/output_v2/image03.txt

Validating files...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.21s/it]

Error! Check file: /data/NDA/output_v2/image03.txt
> /home/XXX/proj/nda-tools/NDATools/Validation.py(138)output()
-> column = v['columnName']
(Pdb) up
> /home/XXX/proj/nda-tools/NDATools/clientscripts/vtcmd.py(170)validate_files()
-> validation.output()
(Pdb) l
165             if response['status'] == Status.SYSERROR:
166                 print('\nSystemError while validating: {}'.format(file))
167                 print('Please contact [email protected]')
168             elif response['errors'] != {}:
169                 print('\nError! Check file: {}'.format(file))
170  ->     validation.output()
171         print('Validation report output to: {}'.format(validation.log_file))
172     
173         if warnings:
174             validation.get_warnings()
175             print('Warnings output to: {}'.format(validation.log_file))
(Pdb) p file
'/data/NDA/output_v2/image03.txt'
(Pdb) down
> /home/XXX/proj/nda-tools/NDATools/Validation.py(138)output()
-> column = v['columnName']
(Pdb) l
133                              'MESSAGE':'None','RECORD': 'None'})
134                     else:
135                         for error, value in response['errors'].items():
136                             for v in value:
137                                 import pdb; pdb.set_trace()
138  ->                             column = v['columnName']
139                                 message = v['message']
140                                 try:
141                                     record = v['recordNumber']
142                                 except KeyError:
143                                     record = ' '
(Pdb) p response
{'status': 'CompleteWithErrors', 'errors': {'data_structure': [{'message': 'File not recognized as a data file, and was not associated or referenced from a data file.'}]}, 'scope': None, 'expiration_date': '2019-06-27T11:12:59.663-0400', 'done': True, 'id': '9fb117d6-9a51-4645-90a5-1db0fe2f4a9b', 'manifests': [], '_links': {'self': {'href': 'https://nda.nih.gov/api/validation/9fb117d6-9a51-4645-90a5-1db0fe2f4a9b'}}, 'warnings': {}, 'associated_file_paths': [], 'short_name': None}
(Pdb) p response['errors']
{'data_structure': [{'message': 'File not recognized as a data file, and was not associated or referenced from a data file.'}]}
(Pdb) p error
'data_structure'
(Pdb) p value
[{'message': 'File not recognized as a data file, and was not associated or referenced from a data file.'}]

as you can see there is only message

Question About Data Types

Hello!

I hope you are well.

I am trying to upload a set of data but keep getting stuck on an error where my experiment ID is a string, when it should be an integer. Is this a new requirement for NDA uploads? In the past, the ID could be a string, or am I wrong?

Thank you for your time!

downloadcmd <packageid> fails with "list index out of range" and exits without error code

$> downloadcmd 1478637
Running NDATools Version 0.2.3
Opening log: /home/yoh/NDAValidationResults/debug_log_20201112T133244.txt
list index out of range

$> echo $?
0

$> cat /home/yoh/NDAValidationResults/debug_log_20201112T133244.txt
2020-11-12 13:32:44,228:INFO:request to generate AWS token

so I guess somewhere it catches an exception and just prints it and happily exists. Should exit with non-0.

It also remains unclear what actually caused the exception and how to mitigate (most likely credentials - either minting a token; IIRC there were some recentish updates?)

NB package id in above oculd have been a wrong paste, but it behaves similarly with another, correct package id 1182919

Not able to use the downloadcmd

Running into the following error trying to run downloadcmd -h

Running NDATools Version 0.2.10
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.8/site-packages/requests/models.py", line 382, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 392, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 2, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://pypi.org/pypi/nda-tools/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/.local/bin/downloadcmd", line 5, in <module>
    from NDATools.clientscripts.downloadcmd import main
  File "/home/me/.local/lib/python3.8/site-packages/NDATools/__init__.py", line 42, in <module>
    check_version()
  File "/home/me/.local/lib/python3.8/site-packages/NDATools/__init__.py", line 22, in check_version
    req = requests.get(url_pattern.format(package=package))
  File "/home/me/.local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/me/.local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/me/.local/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/home/me/.local/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/home/me/.local/lib/python3.8/site-packages/requests/models.py", line 316, in prepare
    self.prepare_url(url, params)
  File "/home/me/.local/lib/python3.8/site-packages/requests/models.py", line 384, in prepare_url
    raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: https://pypi.org/pypi/nda-tools/json

nda-tools should adjust permissions to prevent passwd leakage from ~/.NDATools/settings.cfg

On the systems with default umask of 022, this config file will be created world-readable.
nda-tools should chmod it to 600 if they are creating it

UTF encoding error in api_request

I have been getting the follow error when trying to validate a manifest file submission:

UnicodeEncodeError: 'latin-1' codec can't encode character '\ufeff' in position 0: Body ('\ufeff') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

This can be fixed by adding a call to the .encode() method below line 79 in Utils.py:

79     try:
80         if data is not None:
81             data = data.encode('utf-8')
82         r = session.send(requests.Request(verb, endpoint, headers,           auth=auth, data=data).prepare(),
83                          timeout=300, stream=False)

Happy to put in a PR if that would make things easier.

Seems to require python 3?

downloadcmd crashes on python 2:
Traceback (most recent call last): File "/Users/alex/anaconda/bin/downloadcmd", line 7, in <module> from NDATools.clientscripts.downloadcmd import main File "/Users/alex/anaconda/lib/python2.7/site-packages/NDATools/clientscripts/downloadcmd.py", line 7, in <module> from NDATools.Download import Download File "/Users/alex/anaconda/lib/python2.7/site-packages/NDATools/Download.py", line 94 print(*args, **kwargs) ^ SyntaxError: invalid syntax

works in python 3...

Error with reading password when it has a % character in it.

I just started using this software and ran into this error (all personal information removed. It does show my PPI in the error message and I censored it out.

Running NDATools Version 0.2.0
Opening log: /home/MyUsername/NDAValidationResults/debug_log_20200520T170857.txt
Enter your NIMH Data Archives username:MyUsername
Enter your NIMH Data Archives password:
Enter your aws_access_key. If none, hit "Enter:"
Enter your aws_secret_key. If none, hit "Enter":
Traceback (most recent call last):
  File "/home/MyUsername/MyEnv/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/MyUsername/MyEnv/lib/python3.5/site-packages/NDATools/clientscripts/downloadcmd.py", line 69, in main
    config = configure(args.username, args.password)
  File "/home/MyUsername/MyEnv/lib/python3.5/site-packages/NDATools/clientscripts/downloadcmd.py", line 64, in configure
    config.make_config()
  File "/home/MyUsername/MyEnv/lib/python3.5/site-packages/NDATools/Configuration.py", line 88, in make_config
    copy_config.set("User", "password", self.password)
  File "/cm/shared/c3ddb/python/3.5.2/lib/python3.5/configparser.py", line 1190, in set
    super().set(section, option, value)
  File "/cm/shared/c3ddb/python/3.5.2/lib/python3.5/configparser.py", line 891, in set
    value)
  File "/cm/shared/c3ddb/python/3.5.2/lib/python3.5/configparser.py", line 401, in before_set
    "position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in 'MyPasswordWith%Character' at position 11

This was fixed by logging into the nda website and simply changing my password to something without a %

Where does vtcmd install to?

Even when installation seemed to succeed, I get a "command not found" for vtcmd, and I don't see a folder in any of the obvious places. What am I looking for, and where am I looking for it?

KeyError from mis-parsed URI during Submission

When uploading images and after successful validation of my image03 csv(s), I was getting the following exception:

Traceback (most recent call last):
  File "/home/jacoblee/miniconda3/envs/ndar/bin/vtcmd", line 8, in <module>
    sys.exit(main())
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 442, in main
    submit_package(package_id=package_id, full_file_path=full_file_path,
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 325, in submit_package
    submission.submission_upload(hide_progress=config.hideProgress)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 366, in submission_upload
    self.batch_update_status(status=Status.PROCESSING)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 326, in batch_update_status
    list_data = self.generate_data_for_request(status)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 143, in generate_data_for_request
    size = self.full_file_path[file][1]
KeyError: 'submission_54752/980602.T1I-0000529-001.002-000011-121947.ima'

I ran the code through a debugger and identified the full_file_path dictionary, and the keys were not prefixed with submission_54752/ (the exact number depends on which submission attempt of course). I noticed that the destination_uri in cred was s3://NDAR_Central_2/NDAR_Central_2/submission_54752/980602.T1I-0000529-001.002-000011-121947.ima, and that the code:

file = cred['destination_uri'].split('/')
file = '/'.join(file[4:])

was causing the mismatch. I was able to fix and successfully submit by changing the code to read file[5:].

However, those two lines of code haven't changed in several years, and I was able to submit successfully for another grant only a couple of weeks ago. I can only assume that the destination_uri changed between submissions, with an extra / put in somewhere. I don't think that I did anything differently on my end (I largely re-use the same code).

Download Package Bug?

Hi,

Sorry to hit you with another issue. I was just curious about something I see when I download a package from the NDA website. It looks like the files associated with it download, but instead of the program exiting smoothly, I get an error that says

downloadcmd -dp $1 -d $2 -d $2 -u $3 -p $4

returned with nonzero exit status? All of the variables preceded by a $ sign are my username, password, etc. Is this a known bug that I shouldn't worry about or does it mean that all the files have not downloaded successfully?

Thanks!

John

Version flag to vtcmd

It would be nice to have a basic --version flag to vtcmd to know which version is being used at any time.

format of S3_path_list

I'm trying to download just a subset of S3 links from the datastructure_manifest.txt of a particular package, per the downloadcmd -h instructions pasted here for reference in their entirety:

(abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ downloadcmd -h
Running NDATools Version 0.2.3
Opening log: /home/ubuntu/NDAValidationResults/debug_log_20210122T165559.txt
usage: downloadcmd <S3_path_list>

This application allows you to enter a list of aws S3 paths and will download
the files to your drive in your home folder. Alternatively, you may enter a
packageID, an NDA data structure file or a text file with s3 links, and the
client will download all files from the S3 links listed. Please note, the
maximum transfer limit of data is 5TB at one time.

positional arguments:
<S3_path_list> Will download all S3 files to your local drive

optional arguments:
-h, --help show this help message and exit
-dp, --package Flags to download all S3 files in package.
-t, --txt Flags that a text file has been entered from where to
download S3 files.
-ds, --datastructure Flags that a data structure text file has been entered
from where to download S3 files.
-u , --username
NDA username
-p , --password
NDA password
-r , --resume
Flags to restart a download process. If you already
have some files downloaded, you must enter the
directory where they are saved.
-d , --directory
Enter an alternate full directory path where you would
like your files to be saved.
-wt , --workerThreads
Number of worker threads
-v, --verbose Option to print out more detailed messages as the
program runs.

My S3_list in the command below is a text file that contains a list of S3 links:
s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.json
s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.nii.gz
s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/OTHER_FILES/4088_01_MR_BIAS_64CH.json

**Using such a list results in a downloaded folder of nothing, and a 'list index out of range' message.

Any ideas on what might be missing?**

(abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ downloadcmd S3_list -d HCPEPReleaseDLS3listtest -v -u (myusername) -p (mypassword)

Running NDATools Version 0.2.3
Opening log: /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt
list index out of range
Finished downloading all files.

(abcd3.5.1) ubuntu@ip-172-31-47-9:/spacedir$ less /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt
(abcd3.5.1) ubuntu@ip-172-31-47-9:/spacedir$ cat /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt
2021-01-22 16:48:10,920:INFO:request to generate AWS token
(abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$

Add test(s)

Currently not a single test is provided and thus contributions are hindered by inability to verify correct operation etc.

Please let me know if you need assistance with establishing at least basic testing (e.g. using Travis CI and pytest)

must use a real method to handle secrets

I have declined to install this tool at the NIH HPC on behalf of a user because it does not handle secrets appropriately. In particular it saves usernames and passwords in a plain text file in the user's home directory. If these data are unencrypted at rest I have no reason to believe they are encrypted properly in transit either. Please change to a well-established key, token, or password model for secret management.

Stop forcing us to update

The title says it all. It seems the tool doesn't run if it detects a newer one has been released, forcing us to update. However, updating brings updates to other dependencies which in turn breaks other packages that are more important than nda-tools.

Please, don't force this anymore.

Running NDATools Version 0.2.16
Your version of nda-tools is out of date. Please upgrade to the latest version (0.2.21) from PyPi or GitHub and try again. 
	To upgrade using pip, run: 
pip install nda-tools==0.2.21

Btw, I can't stop getting amazed about how difficult it is to download ABCD and other NDA data. It has been 4 years since I started working with this data and it continues to be clunky, difficult, buggy, with outdated documentation in the NDA website, video tutorials that are no longer valid, and all sorts of issues.

Life is short, we have a lot of things to do, can't waste weeks trying to do something as mundane as downloading data.

Isn't the full time job of somebody to make this data accessible? Why does it have to be so lousy?

Vtcmd minor feature request: Option to exit with error instead of prompt for missing input

I’m working with the latest NDA Tools code. I’m working on a plugin for our webapp and want to write a little wrapper client for vtcmd. For this client, it would be useful if any incorrect/missing parameter information would result in the script throwing an error rather than prompting for input. Would you mind adding an optional parameter to the script to cause the script to exit with an error status rather than prompt for incorrect/missing parameter values?

`KeyError : 'content-type' error`, even when package ID was provided through `-dp`

I used the command,
downloadcmd -dp 12038XX -d '/scratch/08834/tg881334/ABCD_release4' -u 'XXXX' -p 'YYYY'

However, I got the following error, stating that there was a KeyError :

(download_TRY) login4.stampede2(456)$ downloadcmd -dp 12038XX -d '/scratch/08834/tg881334/ABCD_release4' -u 'XXX' -p 'YYYY' 
Running NDATools Version 0.2.18
Warning: Support for the password flag (-p, --password) has been removed from nda-tools due to security concerns and has been replaced with keyring.
The password that was entered for user 'XXXX' is invalid ...
If your username was previously entered incorrectly, you may update it in your settings.cfg located at 
/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/clientscripts/config/settings.cfg
Enter your NIMH Data Archives password:

No value specified for --workerThreads. Using the default option of 27
Important - You can configure the thread count setting using the --workerThreads argument to maximize your download speed.


Getting Package Information...
Traceback (most recent call last):
  File "/home1/08834/tg881334/logout/envs/download_TRY/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
    s3Download.start()
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/Download.py", line 191, in start
    package_resource = self.get_package_info()
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/Download.py", line 952, in get_package_info
    tmp = get_request(url,headers=self.request_header(),auth=self.auth)
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/Utils.py", line 289, in get_request
    error_handler(tmp)
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/NDATools/Utils.py", line 60, in print_and_exit
    if 'json' in r.headers['Content-Type']:
  File "/home1/08834/tg881334/logout/envs/download_TRY/lib/python3.7/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-type'

I looked at previous GitHub issues, and this one : #45 states that this occurs when the package ID was not specified via -dp. However, provided the -dp option. Could anyone help me determine where I did wrong?

Thank you in advance :)

Check for symlinks and option to force derefencing

I'm not sure what the current behavior of vtcmd is with respect to symlinks, but if it does not already deference symlinks when uploading files, it would be a useful feature. Often we have prepared the source files and named the base directory for internal organization in a way that looks messy or confusing for data users and we'd like to be able to use symlinks to manage that.

Adding a Text File to Uploads

Hi Greg!

Is there any way to add a text file with an explanation of some data elements to an NDA upload? I keep getting a validation error when I try to include it, so don't know if it's possible.

Thanks!

John

Trouble downloading file

Hello,

I am trying download files from the NDA, and when using a fresh conda python environment with only nda-tools and secretstorage installed, along with their dependencies. When I attempt to use downloadcmd and a S3 link, it gives the following error message:

downloadcmd s3://NDAR_Central_1/submission_XXXXX/NDARXXXXXX_baselineYear1Arm1_ABCD-SST-fMRI_XXXXXXXXXX.tgz
Running NDATools Version 0.2.21
Warning: Detected non-empty value for "password" in settings.cfg. Support for this setting has been deprecated and will no longer be used by this tool. Password storage is not recommended for security considerations
-u/--username argument not provided. Using default value of 'XXXXX' which was saved in /home/XXXXX/.NDATools/settings.cfg
Enter your NIMH Data Archives password:

No value specified for --workerThreads. Using the default option of 71
Important - You can configure the thread count setting using the --workerThreads argument to maximize your download speed.


Getting Package Information...

An unexpected error was encountered and the program could not continue. Error message from service was: 
Failed to convert value of type 'java.lang.String' to required type 'java.lang.Long'; nested exception is java.lang.NumberFormatException: For input string: "None"


Exit signal received, shutting down...

Any assistance you may be able to provide in the manner would be greatly appreciated. Thank you!

vtcmd fails if log directory doesn't exist

On first time installation, no directory at ${HOME}/NDAValidationResults/ exists to create a log file.

(nda-python3) [kdaily@tigger nda-tools]$ vtcmd -h
Traceback (most recent call last):
  File "/home/kdaily/.virtualenvs/nda-python3/bin/vtcmd", line 6, in <module>
    from NDATools.clientscripts.vtcmd import main
  File "/home/kdaily/.virtualenvs/nda-python3/lib/python3.7/site-packages/NDATools/clientscripts/vtcmd.py", line 2, in <module>
    from NDATools.Validation import Validation
  File "/home/kdaily/.virtualenvs/nda-python3/lib/python3.7/site-packages/NDATools/Validation.py", line 13, in <module>
    from NDATools.Utils import *
  File "/home/kdaily/.virtualenvs/nda-python3/lib/python3.7/site-packages/NDATools/Utils.py", line 16, in <module>
    logging.basicConfig(filename=log_file, level=logging.DEBUG, format="%(asctime)s:%(levelname)s:%(message)s")
  File "/usr/lib64/python3.7/logging/__init__.py", line 1900, in basicConfig
    h = FileHandler(filename, mode)
  File "/usr/lib64/python3.7/logging/__init__.py", line 1092, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python3.7/logging/__init__.py", line 1121, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/home/kdaily/NDAValidationResults/debug_log_20190624T092809.txt'

Subject_01 File Submission

Hello!

I hope you are well! I was recently made aware that NDA wants a "ndar_subject01" file along with the image03.csv file when a new MRI submission is uploaded:

https://nda.nih.gov/data_structure.html?short_name=ndar_subject01

Do you know of any way to upload this file along with the rest of the submission when using NDA-Tools to do the upload?

Thanks!

John

having full local paths in .csv for data files still requires some location to be entered

which makes little sense. I think analysis should be done in code and if paths are absolute, there should be no question asked. So I had to enter / directory as the location:

/data/R01/bids/Brain/wpc-7642/sub-719531/ses-01/func/sub-719531_ses-01_task-recognitionFood_acq-MB8_run-02_bold.nii.gz
/data/R01/bids/Brain/wpc-7642/sub-719531/ses-01/func/sub-719531_ses-01_task-rest_acq-MB8_run-01_bold.nii.gz
Press the "Enter" key to specify directory/directories OR an s3 location by entering -s3 <bucket name> to locate your associated files:/
Building Package

which later on failed anyways with

  File "/home/XXX/proj/nda-tools/NDATools/Submission.py", line 352, in batch_update_status
    list_data = self.generate_data_for_request(status)
  File "/home/XXX/proj/nda-tools/NDATools/Submission.py", line 132, in generate_data_for_request
    size = self.full_file_path[file][1]
KeyError: 'data/NDA/output_v2/sub-719218_ses-01_task-rest_acq-MB8_run-01_bold.metadata.zip'

support full paths in the file list .csv

related to #14

ATM, as shown in #14 traceback:

  File "/home/XXX/proj/nda-tools/NDATools/Submission.py", line 352, in batch_update_status
    list_data = self.generate_data_for_request(status)
  File "/home/XXX/proj/nda-tools/NDATools/Submission.py", line 132, in generate_data_for_request
    size = self.full_file_path[file][1]
KeyError: 'data/NDA/output_v2/sub-719218_ses-01_task-rest_acq-MB8_run-01_bold.metadata.zip'

full paths were generated by bids2nda tool, the issue to request storing relative paths is filed: bids-standard/bids2nda#19

downloadcmd not working properly, generates several errors when trying to log in

Seeing the issues below when trying to run downloadcmd

Enter your NIMH Data Archives username:
Traceback (most recent call last):
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/util.py", line 48, in send_and_get_reply
    raise DBusErrorResponse(resp_msg)
jeepney.wrappers.DBusErrorResponse: [org.freedesktop.DBus.Error.UnknownMethod] ("No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/secrets/collection/login",)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/collection.py", line 177, in get_default_collection
    return Collection(connection)
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/collection.py", line 45, in init
    self._collection.get_property('Label')
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/util.py", line 67, in get_property
    (signature, value), = self.send_and_get_reply(msg)
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/util.py", line 52, in send_and_get_reply
    raise ItemNotFoundException('Item does not exist!') from resp
secretstorage.exceptions.ItemNotFoundException: Item does not exist!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/software/nda-tools/0.2.21-py-3.8/keyring/backends/SecretService.py", line 61, in get_preferred_collection
    collection = secretstorage.get_default_collection(bus)
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/collection.py", line 179, in get_default_collection
    return create_collection(connection, 'Default', 'default', session)
  File "/software/nda-tools/0.2.21-py-3.8/secretstorage/collection.py", line 159, in create_collection
    raise PromptDismissedException('Prompt dismissed.')
secretstorage.exceptions.PromptDismissedException: Prompt dismissed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/software/nda-tools/0.2.21-py-3.8/bin/downloadcmd", line 8, in
    sys.exit(main())
  File "/software/nda-tools/0.2.21-py-3.8/NDATools/clientscripts/downloadcmd.py", line 182, in main
    config = configure(args)
  File "/software/nda-tools/0.2.21-py-3.8/NDATools/clientscripts/downloadcmd.py", line 172, in configure
    config.read_user_credentials()
  File "/software/nda-tools/0.2.21-py-3.8/NDATools/Configuration.py", line 162, in read_user_credentials
    self.password = keyring.get_password(self.SERVICE_NAME, self.username)
  File "/software/nda-tools/0.2.21-py-3.8/keyring/core.py", line 55, in get_password
    return get_keyring().get_password(service_name, username)
  File "/software/nda-tools/0.2.21-py-3.8/keyring/backends/SecretService.py", line 78, in get_password
    collection = self.get_preferred_collection()
  File "/software/nda-tools/0.2.21-py-3.8/keyring/backends/SecretService.py", line 63, in get_preferred_collection
    raise InitError("Failed to create the collection: %s." % e)
keyring.errors.InitError: Failed to create the collection: Prompt dismissed..

Invalid password error

Hello NDA team,

I am getting an invalid password error though I am using the exact credentials for logging successfully into NDA login.gov account. I have also setup keyring and I can see my password when I run keyring.get_password("nda-tools", "username"). does nda-tools have access to login.gov account? thanks

Is authentication required for validation?

Previously, we don't need to provide username and password for validation, but this recent change suggests otherwise.

nda-tools/NDATools/Validation.py

Line 61 in 16e8c26

self.auth = requests.auth.HTTPBasicAuth(self.config.username, self.config.password)

Import readline module for better text entry

In vtcmd, arrow keys do not work correctly. In Python3, you can resolve this by importing readline, which will allow standard shell behaviors before the input is loaded into the program. e.g.,

thing = input('>>>')

Followed by ACB<-<-, will result in ACB[[A[[A| (with | the cursor location) on some systems.
However,

import readline
thing = input('>>>')

Followed by the same will result in the desired A|CB.

Question: GUID generation

I'm getting some pushback on installing/updating java and this tool was suggested as an alternative to the NDAR java web-services.

Unfortunately, I don't see a GUID creation tool. Did I miss it? are there any plans to provide one?

Thanks in advance.

git tag the releases

there are versions in CHANGELOG but not a single git tag.

Note: better git tag -a locally, since github doesn't generate annotated tags for the releases

Vtcmd feature request: Add option to hide progress monitor

I've run into failures relating to the progress monitor in the Linux environment. Vtcmd submissions work fine from the command line but fail when launched from Java's Runtime environment. The issue could be fixed by setting the "ascii" option to true for all operating systems (currently it's "ascii=os.name == 'nt'"), however it would be preferable to have an option to hide the progress monitors. I've coded the option for our internal use, along with some improvements to outcome reporting when hiding the progress monitor, here:

https://github.com/hodgem/nda-tools/commits/ccf_customizations

I'm thinking it might be useful to others to have this option as well. Thanks!

NDA Downloads/File Deletions

Hello again!

Sorry to bother you with yet another question.

If I would like to download data from the NDA website using the commands given in the ReadMe section of NDA-Tools, do I need an AWS account? It looks like that is the best way. But wanted to ask.

Also, if I ever want to delete a previously uploaded file (for instance, it is a duplicate of another uploaded file), is there a way to do this, as well?

Thanks again for all your help,

John

Rename cmdline tools with consistent (and unique) prefix

I would strongly recommend to rename downloadcmd into something not that generic. With that in mind I would recommend to rename them all to have nda- prefix, e.g.

• nda-download (instead of generic downloadcmd)
• nda-upload (instead of cryptic vtcmd)

error after upload has completed

I recevied the folowing error after an upload had completed, from the error im left unsure if the Submission is now imcomplete:

Uploads complete.
Checking Submission Status.
There was an error transferring some files, trying again
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/vtcmd", line 10, in
sys.exit(main())
File "/home/ubuntu/.local/lib/python2.7/site-packages/NDATools/clientscripts/vtcmd.py", line 290, in main
threads=args.workerThreads, batch=args.batch, config=config)
File "/home/ubuntu/.local/lib/python2.7/site-packages/NDATools/clientscripts/vtcmd.py", line 258, in submit_package
submission.submission_upload(hide_progress=config.hideProgress)
File "/home/ubuntu/.local/lib/python2.7/site-packages/NDATools/Submission.py", line 457, in submission_upload
self.submission_upload()
TypeError: submission_upload() takes exactly 2 arguments (1 given)

Error with vtcmd --warning

Running vtcmd with the -w argument fails to execute properly, see below.

vtcmd genomics_subject02.csv nichd_btb02.csv genomics_sample03.csv -w

Traceback (most recent call last):
  File "/users/swengcho/.conda/envs/cloud/bin/vtcmd", line 10, in <module>
    sys.exit(main())
  File "/users/swengcho/.conda/envs/cloud/lib/python3.7/site-packages/NDATools/clientscripts/vtcmd.py", line 257, in main
    validation_results = validate_files(args.files, w, bp, threads=args.workerThreads, config=config)
  File "/users/swengcho/.conda/envs/cloud/lib/python3.7/site-packages/NDATools/clientscripts/vtcmd.py", line 160, in validate_files
    validation.warnings()
AttributeError: 'Validation' object has no attribute 'warnings'

The issue is traceable here;

nda-tools/nda-tools-python/NDATools/clientscripts/vtcmd.py

Line 160 in 4a8461e

validation.warnings()

Based on the validation object defined by Validation.py, this should be validation.get_warnings()

KeyError

hi i tried to download a single file using example command downloadcmd s3://NDAR_Central_3/submission_33230/HCD0008117_V1_MR/T1w/T1w_acpc_dc_restore.nii.gz -u <your username> -p <your password> -d HCPDevImgManifestBehSingleLink from
https://www.humanconnectome.org/storage/app/media/documentation/LS2.0/LS_Release_2.0_Access_Instructions_June2022.pdf and i got the error below.

Running NDATools Version 0.2.17

No value specified for --workerThreads. Using the default option of 31
Important - You can configure the thread count setting using the --workerThreads argument to maximize your download speed.


Getting Package Information...
Traceback (most recent call last):
  File "/home/junhong.yu/NDAenv/bin/downloadcmd", line 10, in <module>
    sys.exit(main())
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/NDATools/clientscripts/downloadcmd.py", line 197, in main
    s3Download.start()
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/NDATools/Download.py", line 191, in start
    package_resource = self.get_package_info()
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/NDATools/Download.py", line 952, in get_package_info
    tmp = get_request(url,headers=self.request_header(),auth=self.auth)
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/NDATools/Utils.py", line 289, in get_request
    error_handler(tmp)
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/NDATools/Utils.py", line 60, in print_and_exit
    if 'json' in r.headers['Content-Type']:
  File "/home/junhong.yu/NDAenv/lib/python3.7/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-type'