mitre / fusera

A FUSE interface to the NCBI Sequence Read Archive (SRA)

Software Tools

fusera

fusera (FUSE for SRA) is a FUSE implementation for the cloud extension to the NCBI Sequence Read Archive (SRA). SRA accepts data from all kinds of sequencing projects including studies that involve human subjects or their metagenomes, which may contain human sequences. Access to this data is controlled via dbGaP (the database of Genotypes and Phenotypes). To gain access to data from such a controlled access study, users would submit a Data Access Request (DAR) for their research project. Users with approved DARs can access the study's phenotype and genotype data via dbGaP, and the sequence data via the SRA. Traditionally, the SRA was a repository that facilitated download of data to the user's compute environment. As part of the Sequence Data Delivery Pilot (SDDP), the SRA has been extended to provide access to cloud-hosted data through its Data Locator web-services API using signedURL. fusera presents selected SRA data referenced with an SRA accession number to authorized users as a read-only file system, enabling users and tools to access the data through a file system interface.

fusera presents all of the cloud-hosted SRA data for a set of SRA Accession numbers as a mounted directory, with one subdirectory per SRA Accession number. The user’s credentials are passed through a dbGaP repository key, or ngc file, that is obtained from dbGaP. fusera automatically detects the cloud region where it is deployed on both AWS and GCP, and requests access to data from the SRA's Data Locator web-servvices API within that same region with the aim of limiting charges for data egress. fusera is intended for deployment on AWS or GCP.

sracp

Users may want to copy data from its cloud bucket into the local storage of their cloud-hosted virtual machine, instead of operating on the data in-place using fusera. sracp addresses this need. Given a list of SRA accession numbers, a dbGaP repository key, and a target directory, sracp copies the data from its cloud bucket into a specified target directory on a cloud-based virtual machine. The user’s credentials are passed through a dbGaP repository key, or ngc file, that is obtained from dbGaP. sracp automatically detects the cloud region where it is deployed on both AWS and GCP, and requests access to data from the SRA's Data Locator web-servvices API within that same region with the aim of limiting charges for data egress. sracp is intended for deployment on AWS or GCP.

Installation

See installation instructions on the wiki.

Dependencies

fusera

Depending on the linux distro, fuse-utils may need to be installed.

Mac users must install osxfuse either on their website or through Homebrew:

brew cask install osxfuse

Usage

In an effort to keep the instructions as up to date as possible, please refer to the wiki for instructions on how to use fusera and sracp.

Troubleshooting

See troubleshooting page on the wiki.

License

fusera started its life as a hard fork of the Goofys project.

Copyright (C) 2015 - 2017 Ka-Hing Cheung

Modifications Copyright (C) 2018 The MITRE Corporation

The modifications were developed for the NIH Data Commons Pilot. General questions can be forwarded to:

[email protected]
Technology Transfer Office
The MITRE Corporation
7515 Colshire Drive
McLean, VA 22102-7539

Licensed under the Apache License, Version 2.0

Only the functionality needed was retained from the Goofys project. These files were removed from the original source:

• api/api.go
• internal/
  • perms.go
  • ticket.go
  • ticket_test.go
  • v2signer.go
  • minio_test.go
  • goofys_test.go
  • aws_test.go

There has also been some refactoring of the codebase, so while some files have been removed, the code in them might exist in other files. License headers and copyright have been kept in these circumstances.

The major changes to the original source stem from fusera's use case. Goofys was designed to present the content of a single bucket as a file system. fusera is designed to present the data associated with a collection of SRA accessions as a file system, and the data can be spread across many buckets.

Goofys' start up was modified to determine the content of the presented file system based on querying the NCBI Data Locator API, rather than querying the content of the target bucket.

References

fusera owes much to the Goofys project: a high-performance, POSIX-ish file system written in Go which was used as a starting point.

fusera's Issues

Fusera and scarp should include binary name and version in http requests to SDL

In order to assist SDL logging, add binary name and version to http requests (User Agent) instead of sending the default (go-http).

Add a retry to the batch requests in the nr package

If the nr package receives an error that is related to the network or the server, try again in order to see if the API was just having an issue.

Long list of Accessions in command line --> error

/home/ubuntu/bin/fusera mount -n /home/ubuntu/ngc.ngc -a SRR6778594,SRR6776943,SRR6779406,SRR6777798,SRR6778169,SRR6776868,SRR6778714,SRR6777907,SRR6777913,SRR6777652,SRR6779035,SRR6777207,SRR6779177,SRR6777840,SRR6778171,SRR6777181,SRR6776977,SRR6776627,SRR6778604,SRR6777005,SRR6778882,SRR6778139,SRR6776962 -l s3.us-east-1 /home/ubuntu/fusemnt &

Works fine but adding one more accession (SRR6778881) goes to show

Error: couldn't open cart file at: SRR6778594,SRR6776943,SRR6779406,SRR
6777798,SRR6778169,SRR6776868,SRR6778714,SRR6777907,SRR6777913,SRR6777652,SRR6779035,SRR6777207,S
RR6779177,SRR6777840,SRR6778171,SRR6777181,SRR6776977,SRR6776627,SRR6778604,SRR6777005,SRR6778882
,SRR6778139,SRR6776962,SRR6778881: open SRR6778594,SRR6776943,SRR6779406,SRR6777798,SRR6778169,SR
R6776868,SRR6778714,SRR6777907,SRR6777913,SRR6777652,SRR6779035,SRR6777207,SRR6779177,SRR6777840,
SRR6778171,SRR6777181,SRR6776977,SRR6776627,SRR6778604,SRR6777005,SRR6778882,SRR6778139,SRR677696
2,SRR6778881: file name too long
Usage:
fusera mount [flags] /path/to/mountpoint
…

BTW:
/home/ubuntu/bin/fusera mount -n /home/ubuntu/ngc.ngc -a ./acc.txt -l s3.us-east-1 /home/ubuntu/fusemnt &
works fine.
ngc.ngc is prj_8001.ngc

fusera help should print version number

fusera help should print version number .
As of 0.0.14 it doesn't.

Fusera shows no folders when location is faulty

If an incorrect location is given, no folders are built. Fusera should probably at least build a folder and include that there were no files for that accession in this location in the error.log to help users understand that they gave an incorrect location.

Documentation: Room for confusion about fusera startup

If fusera is started in the foreground, confusion sometimes results, since users may think it has hung.
Documentation should clearly state that fusera should be started in the background, and link to the shell script in https://github.com/mitre/fusera/wiki/Scripting-with-Fusera for an example. #

Add a fusera docker image to docker hub

Add eager flag that signs urls on start up

correct the help printout where it incorrectly states the batch defaults are 0

running with -a and a large list of accessions (5077) doesn't work

fusera mount -a PublicAccessAccessions5077.txt ./fusemnt &
PublicAccessAccessions5077.txt

time ~/bin/fusera.sh mount -a PublicAccessAccessions5077.txt ~/fusemnt_all
waiting for /home/ubuntu/fusemnt_all/.initialized
/home/ubuntu/fusemnt_all initialized!

real 0m31.031s
user 0m0.000s
sys 0m0.004s
ubuntu@ip-172-31-43-159:~$ ls -lR fusemnt_all
fusemnt_all:
total 0

fusera 0.0.8 and CWL

From Todd Pihl @ NCI

==============================
Sorry, v0.0.8 gives the same result:

cwl-runner mountfusera.yml mountthis.yml
/usr/local/bin/cwl-runner 1.0.20180622214234
Resolved 'mountfusera.yml' to 'file:///home/ubuntu/datalake/mountfusera.yml'
[job mountfusera.yml] /tmp/tmpjoetGB$ /home/ubuntu/bin/fusera
mount
--ngc
/home/ubuntu/datalake/prj_phs710EA_test.ngc
--accession
SRR2043622
--location
s3.us-east-1
/home/ubuntu/datalake/sddpmount
panic: user: Current not implemented on linux/amd64

goroutine 1 [running]:
github.com/mitre/fusera/cmd.myUserAndGroup(0x7fff438c0f05, 0x1f)
/Users/mbianchi/src/github.com/mitre/fusera/cmd/mount.go:219 +0x295
github.com/mitre/fusera/cmd.mount(0xe55c00, 0xc4201b03f0, 0x1, 0x7, 0x0, 0x0)
/Users/mbianchi/src/github.com/mitre/fusera/cmd/mount.go:146 +0x171
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).execute(0xe55c00, 0xc4201b0380, 0x7, 0x7, 0xe55c00, 0xc4201b0380)
/Users/mbianchi/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:762 +0x475
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xe55e60, 0xc420058058, 0x0, 0x4dcd27)
/Users/mbianchi/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:852 +0x334
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).Execute(0xe55e60, 0x0, 0x0)
/Users/mbianchi/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mitre/fusera/cmd.Execute()
/Users/mbianchi/src/github.com/mitre/fusera/cmd/root.go:51 +0x2d
main.main()
/Users/mbianchi/src/github.com/mitre/fusera/main.go:36 +0x20
[job mountfusera.yml] completed permanentFail
{}
Final process status is permanentFail

========================
Fusera is version 0.0.7 and I’m running an Ubuntu 16.04.4 VM on AWS.

If you want to look at the CWL I’ve used, I’ve got it in my github repo: https://github.com/pihltd/datalake

mountfusera.yml tries to mount using a full command line approach, mountfusera_envvar.yml tries to use environment variables when possible. Both give the same error. mountthis.yml is the file with the variable values.

Both are run using cwl-runner 1.0. 20180622214234

$ cwl-runner mountfusera.yml mountthis.yml or $ cwl-runner mountfusera_envvar.yml mountthis.yml

Both return this error:

cwl-runner mountfusera.yml mountthis.yml
/usr/local/bin/cwl-runner 1.0.20180622214234
Resolved 'mountfusera.yml' to 'file:///home/ubuntu/datalake/mountfusera.yml'
[job mountfusera.yml] /tmp/tmpcNRAhV$ /home/ubuntu/bin/fusera
mount
--ngc
/home/ubuntu/datalake/prj_phs710EA_test.ngc
--accession
SRR2043622
--location
s3.us-east-1
/home/ubuntu/datalake/sddpmount
panic: user: Current not implemented on linux/amd64

goroutine 1 [running]:
github.com/mitre/fusera/cmd.myUserAndGroup(0x7ffd9c1f4f05, 0x1f)
/Users/travis/gopath/src/github.com/mitre/fusera/cmd/mount.go:194 +0x27e
github.com/mitre/fusera/cmd.mount(0xe282e0, 0xc4201d63f0, 0x1, 0x7, 0x0, 0x0)
/Users/travis/gopath/src/github.com/mitre/fusera/cmd/mount.go:143 +0x148
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).execute(0xe282e0, 0xc4201d6380, 0x7, 0x7, 0xe282e0, 0xc4201d6380)
/Users/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xe28540, 0x403dfc, 0xc42006e058, 0x0)
/Users/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).Execute(0xe28540, 0x0, 0x13)
/Users/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mitre/fusera/cmd.Execute()
/Users/travis/gopath/src/github.com/mitre/fusera/cmd/root.go:51 +0x2d
main.main()
/Users/travis/gopath/src/github.com/mitre/fusera/main.go:35 +0x20
[job mountfusera.yml] completed permanentFail
{}
Final process status is permanentFail

The part I think is coming from fusera is starts with the “panic” line and the following pointers to various fusera files. Like I said earlier, from a bash command line, fusera works fine using exactly the same parameters as the cwl scripts. There’s just something about running from cwl that’s causing a hiccup.

Change location resolution to read the location from a file on GCP

sracp needs to stop using the external cURL tool to download files

As a quick solution, sracp passes the task of downloading files to the cURL tool installed on the local machine. This is an added dependency that creates an extra point of failure for sracp that might be confusing to users. We need to either utilize a suitable cURL library written in Go (no cGo links to actual libcurl) or simply write the download code ourselves.

• Research libraries that download files quickly and efficiently
• Replace use of cURL binary with library

-ngc does not work (-n works)

This works:
fusera mount -n prj_phs710EA_test.ngc -a SRR1219805 ./fusemnt

This fails:
fusera mount -ngc prj_phs710EA_test.ngc -a SRR1219805 ./fusemnt
Error: accepts 1 arg(s), received 2
Usage:
fusera mount [flags] /path/to/mountpoint

Fusera should stop attempting to validate cloud regions in loc flag, as these could change.

Fusera doesn't allow urls to s3 or gcp for acc-file location

I believe I overlooked this requirement when allowing ngc files to be located on the cloud. The requirement also wanted it possible to point the acc-file flag to a cloud location and have fusera accept it.

Fusera still mounts if all accessions fail

Include the accessions that failed in a batch request when getting a 500 internal server error

Right now, the nr library simply prints the 500 internal server error received from the API. It would be helpful to have it print out the accessions that failed with that response in order for the user to see which ones were not successful.

have unmount command only attempt to unmount fusera by checking for .initialized file

Add comparison to SRA toolkit to documentation

For all users, the standard has been to use the SRA toolkit. It would be great to have front-and-center documentation about how fusera differs, when fusera is preferred and why, and some use case translation from the sra toolkit to fusera (ie., how would one do a fastq-dump with fusera vs fastq-dump).

Add details of what data are available via fusera to documentation?

I probably missed it in the documentation, but what SRA subsets are available? That is, what accessions are available? Finally, what would be the incantation to mount all available SRA accessions?

failed to locate accessions: API returned no accessions

Hi, here is the error message I get when trying to access sra files from dbGaP. Does this error come from my side?

[gv40@ip-10-9-0-17 ~]$ echo $DBGAP_LOCATION
s3.us-east-1
[gv40@ip-10-9-0-17 ~]$ fusera mount -a "cart_prj17011_201807171153.krt" -d ~/mnt
Error: failed to locate accessions: API returned no accessions
Usage:
fusera mount [flags] /path/to/mountpoint

Flags:
-a, --accession string A list of accessions to mount or path to cart file. ["SRR123,SRR456" | local/cart/file | https://..s3.amazonaws.com/<cart/file>].
Environment Variable: [$DBGAP_ACCESSION]
--aws-batch int ADVANCED: Adjust the amount of accessions put in one request to the SDL API when using an AWS location.
Environment Variable: [$DBGAP_AWS-BATCH] (default 50)
--eager ADVANCED: Have fusera request that urls be signed by the API on start up.
Environment Variable: [$DBGAP_EAGER]
-e, --endpoint string ADVANCED: Change the endpoint used to communicate with SDL API.
Environment Variable: [$DBGAP_ENDPOINT] (default "https://www.ncbi.nlm.nih.gov/Traces/sdl/1/retrieve")
-f, --filetype string comma separated list of the only file types to copy.
Environment Varible: [$DBGAP_FILETYPE]
--gcp-batch int ADVANCED: Adjust the amount of accessions put in one request to the SDL API when using a GCP location.
Environment Variable: [$DBGAP_GCP-BATCH] (default 25)
-h, --help help for mount
-l, --location string Cloud provider and region where files should be located: [cloud.region].
Environment Variable: [$DBGAP_LOCATION]
-n, --ngc string A path to an ngc file used to authorize access to accessions in DBGaP: [local/ngc/file | https://..s3.amazonaws.com/<ngc/file>].
Environment Variable: [$DBGAP_NGC]

Global Flags:
-d, --debug Enable debug output.

DEBUG 2018/07/17 15:56:01 errors.go:74: failed to locate accessions: API returned no accessions
Failed to locate accessions: It seems that Fusera has encountered an error while using the SRA Data Locator API to determine the file locations for accessions. This is an issue between Fusera and the API. In order to get more information, run Fusera with debug enabled and contact your IT administrator with its contents.

Fusera in Daemon Mode

enable fusera to launch in the daemon mode (FUSE framework does allow this). When it does, we need to be careful in replacing messaging through stdout/stderr with more robust log file.

fusera with native URLs

Currently, fusera assumes that URLs for object incorporate authorization (signedURLs). Significant new data will be open access, and will be sitting in requester pays buckets. In that scenario, fusera must present the user's credentials to S3/GCS to so that charges can be assigned to the user, and not to the SDDP production account.

sracp should check that target location is writeable, and fail with error if not

Create a .stat file that contains the statistics of the mounted file system

This file would exist in the root directory along with the .initialized file as a .stat file and contain useful statistics of the file system such as the total amount of data it represents, how many files it contains, how many folders, etc. A simple read of this file gives the user a gauge of the system as a whole.

Ask for identity when dealing with ftp urls to avoid gzipping

fusera and sracp should have diagnostic mode

Both tools should have a diagnostic mode that access the SDL API to retrieve an unsigned URL and then uses the URL to retrieve data. This will ensure that user's environment isn't in a VPC.
For example, 1000 genomes SRR1219804.

fusera 0.0.15 Segementation Violation

From Adam Tebbe [email protected]:

[ec2-user@ip-10-66-48-122 ~]$ fusera mount --token /home/ec2-user/prj_16421.ngc --accession "SRR8003979" --location s3.us-east-1 ~/studies -v
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x95bc7d]

goroutine 1 [running]:
github.com/mitre/fusera/cmd.mount(0xfe5420, 0xc00002e400, 0x1, 0x8, 0x0, 0x0)
/home/travis/gopath/src/github.com/mitre/fusera/cmd/mount.go:158 +0x1cd
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).execute(0xfe5420, 0xc00002e380, 0x8, 0x8, 0xfe5420, 0xc00002e380)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:762 +0x473
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xfe5680, 0xc0001b7f88, 0x407440, 0xc000094058)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).Execute(0xfe5680, 0x0, 0x0)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mitre/fusera/cmd.Execute()
/home/travis/gopath/src/github.com/mitre/fusera/cmd/root.go:63 +0x2d
main.main()
/home/travis/gopath/src/github.com/mitre/fusera/main.go:36 +0x20

Check that file urls are valid by using them to perform HTTP HEAD calls

The Name Resolver API depends on everything being in order with accounts and permissions on various cloud providers in order to properly sign urls. This means that sometimes, the API can give signed urls that look valid, but don't actually contain the proper permissions to read the file.

Considering this could happen, it might be a good idea to "test" the urls by making an HTTP HEAD request with each url. This will return an error if the permissions for the url are incorrect, but will return the attributes of the file if the url is good. We could continue to validate the API's response by using the attributes we get and comparing them to what the API has given us.

Should we correct if the cloud provider tells us different? I'd say yes, but I also feel like this should be reported? But I'm never a fan of "reporting" an issue and then carrying on like nothing happened.

Fusera still mounts when there is an issue with the folder it's trying to mount to

It appears that a bug has been introduced with the feature of error.log files. Instead of not mounting the file system, fusera lets folder issue errors pass by and mounts anyway. This is because it's supposed to mount if there are some errors with the accessions but definitely not if there's an issue with the folder. This needs to be corrected.

customizing batch not actually working

It seems like fusera is ignoring the batch flags and using the defaults instead

Conduct a refactoring of fusera

• most libraries no longer log
• cleaner error passing between libraries
• debug isn't so noisy
• cleaner error messages to the user
• fusera evaluates the condition of the mount point before trying to resolve a location or make a request to the SDL API

fusera (and sracp?) should issue warning if run as 'root'

sracp progress printout is some times inaccurate

Seems like if there's only one file and there's a failure, the progress says it succeeds and then no file has been copied.

Add tests around fusera to encourage contributing and code quality

Better user experience with fusera's output

1. Create a silent flag that ensures fusera prints nothing for users who want to script around fusera
2. Add a "running" message when fusera successfully mounts (when silent isn't on) to aid interactive users in knowing when fusera has mounted and can be used.
3. Create a verbose flag that prints everything useful for troubleshooting issues and generating a good report for filing issues with fusera

• Print out values of flags
• Print out request fusera makes to SDL API
• Print out response fusera gets from SDL API

Make an easy one step way of installing fusera and sracp

update fusera and sracp to use type query argument with the Name Resolver API

Make fusera and sracp installable through package managers like apt-get.

Amazon Machine Image for fusera?

Is there an AMI that has fusera installed?

Swap the use of urfave/cli with cobra for parsing command line flags/args and env vars

urfave/cli is a good library, but lacks enough documentation to make its use easy to learn. Plus, there seems to be some issues with how printing help works with subcommands and there isn't much explaining how to use subcommands with urfave/cli in the first place. Many tools with subcommands use cobra which gives us plenty of examples to work with, plus the documentation is much more guided and explanatory.

Another added benefit of using cobra is that its design would allow sracp and fusera to cleanly "share" commands, which they do share loc, acc, acc-file, and ngc so this would be a nice code quality "boon".

fusera and sracp need to both understand and handle gzipped data appropriately

When using ftp-ncbi region, the Name Resolver API gives ftp urls. fusera and sracp can still use the http protocol with these urls, but the server handling the requests gzips the data, making it illegible for tools expecting it to be in a raw format.

Keeping it simple

The first solution is a bit primitive, but it seems like the server should respect an http request that specifies an Accept-Encoding of identity which would prevent the server from sending gzipped data in the first place. We would not get to use gzipped data over the wire, but it is still up in the air as to whether gzipped data would even be beneficial in our use case. With fusera, there are likely to be generally small reads which translate to small requests over the wire. With small byte ranges, it may take longer to gzip and decompress the data than it would to send it over the wire raw. But with sracp, the use case is obviously advantageous. sracp wants the whole file, thus a gzip transport of large chunks of generally large files would be beneficial.

A more sophisticated approach

Another more sophisticated and more difficult solution would be if we could recognize that the data is gzipped (through content-type in the header) and decompress it on the fly so that sracp and fusera are none the wiser as to what form the data is in as it comes over the wire. I do see some challenges to overcome when implementing this solution though:

1. When something performs a read of a file with fusera, it gives a byte range it wants. This then results in the http request being made for that range. If the data is gzipped, is the range of raw data gzipped or do we get a byte range of the gzipped data, which will expand to something larger than the range the read requested. What do we do with the extra bytes? Drop them? Hopefully the server gzips on the fly and so the expanded bytes are equivalent to what is requested, making this challenge go away.
2. Can we even decompress gzipped data on the fly? I imagine so, but I believe some decompression protocols need all the compressed data in order to decompress. This concern is born more through my ignorance of the specific way gzip is implemented and so it might be a non-issue.

sracp 0.0.15 segmentation violation

From: Kochergin, Andrey (NIH/NLM/NCBI) [C] [email protected]

Hi,

Does sracp relies on fusemount and fuse?
What would be the correct usage of sracp 0.0.15.?

I am getting error
“
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x93f6c1]

goroutine 1 [running]:
github.com/mitre/fusera/sracp/cmd.glob..func1(0xf9d0e0, 0xc000168410, 0x1, 0xd, 0x0, 0x0)
/home/travis/gopath/src/github.com/mitre/fusera/sracp/cmd/root.go:140 +0x181
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).execute(0xf9d0e0, 0xc00001e0f0, 0xd, 0xd, 0xf9d0e0, 0xc00001e0f0)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:762 +0x473
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xf9d0e0, 0x9, 0x9, 0xc00019bf78)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/mitre/fusera/vendor/github.com/spf13/cobra.(*Command).Execute(0xf9d0e0, 0x5, 0xc000070001)
/home/travis/gopath/src/github.com/mitre/fusera/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mitre/fusera/sracp/cmd.Execute()
/home/travis/gopath/src/github.com/mitre/fusera/sracp/cmd/root.go:269 +0x2d
main.main()
/home/travis/gopath/src/github.com/mitre/fusera/sracp/main.go:33 +0x25
”
in all possible parameter combinations I’ve tried e.g.

sracp -e "https://www.ncbi.nlm.nih.gov/Traces/sdl_test/1/retrieve" -t token -l gs.us -a acc.list -f vcf ./
sracp -e "https://www.ncbi.nlm.nih.gov/Traces/sdl_test/1/retrieve" -t token -l gs.us -a SRR6810109 -f vcf ./

, where token is a file with a jwt and acc.list is a file with 25 accessions parsed from the token
And fusera mounts just fine with the same token

Best regards,
-Andrey

https://www.ncbi.nlm.nih.gov/Traces/sdl/1/locality?acc=SRR6810109

Combine acc-file and acc flags into one accession flag

Right now, there's a flag for passing a path to a cart file and a way to provide a list of accessions. This should be combined into one flag that can recognize a file path or file from a list of accessions so that the options are binary.

document how to launch fusera through vm orchestration on GCP and AWS

Add live error logs for file read errors

fusera does not account for the expiration date after the first expiration happens

Wanted to report a possible bug. After the first set of links expire, fusera does not use the new expiration date but instead keeps requesting the signed urls every time after the first expiration happens.

Update fusera to use the new Name Resolver API features

• Can get file metadata without signing urls (meta-only=yes)
• Lazy signing urls default, add eager flag to do things the old way
• Add filetype flag to fusera to pass on to API in order to only get those filetypes
• Adjust code to read the new error messages from API for cleaner code
• Pass filetype flag from sracp right on to API

document issues with docker and fuse