Serves a web interface and RESTful API for reference genome assets.
Home Page: http://refgenie.databio.org
License: BSD 2-Clause "Simplified" License
A standardized reference genome resource manager. See the documentation.
new YAML file should be created for the server use after refgenieserver archive rather than the original one edited.
location: where the genome_archive key points
use logging (possibly logmuse) instead of warnings in refgenieserver archive
Separate from #32; if I give the config then I get this error:
refgenieserver -c $REFGENIE archive
email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
Traceback (most recent call last):
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 31, in __getattr__
v = super(PathExAttMap, self).__getattribute__(item)
AttributeError: 'YacAttMap' object has no attribute 'set_default'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/ordattmap.py", line 45, in __getitem__
return super(OrdAttMap, self).__getitem__(item)
KeyError: 'set_default'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 34, in __getattr__
return self.__getitem__(item, expand)
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 51, in __getitem__
v = super(PathExAttMap, self).__getitem__(item)
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/ordattmap.py", line 47, in __getitem__
return AttMap.__getitem__(self, item)
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/attmap.py", line 32, in __getitem__
return self.__dict__[item]
KeyError: 'set_default'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nsheff/.local/bin/refgenieserver", line 10, in <module>
sys.exit(main())
File "/home/nsheff/.local/lib/python3.7/site-packages/refgenieserver/main.py", line 124, in main
archive(rgc, args)
File "/home/nsheff/.local/lib/python3.7/site-packages/refgenieserver/server_builder.py", line 52, in archive
asset_desc = rgc[CFG_GENOMES_KEY][genome][CFG_ASSETS_KEY][asset_name].set_default(CFG_ASSET_DESC_KEY, "NA")
File "/home/nsheff/miniconda3/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 38, in __getattr__
raise AttributeError(item)
AttributeError: set_default
I originally used refgenie_server as the name of this repository, and then used refgenies as the command to invoke to start the server.
I'm not sure if refgenies is good because it is just so close to refgenie that it may cause some confusion.
I wanted to see what you thought about this and if you had any other suggestion for CLI name.
some ideas:
refgenservergserverrgcserverrefgenieserverI'm leaning toward rgserver for the command name....
docker build -t fastapi .
docker run --rm -p 80:80 --name fastapi -v $(pwd):/app -v $(pwd)/files:/genomes fastapi refgenieserver -c refgenie.yaml serve
INFO: Started server process [1]
INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
asgi = app(self.scope)
TypeError: __call__() missing 2 required positional arguments: 'receive' and 'send'
INFO: ('172.17.0.1', 35496) - "GET / HTTP/1.1" 500
ERROR: Exception in ASGI application
Will we want to be able to return server metadata for a genome that is requested by hash value?
Some reading:
https://stackoverflow.com/questions/389169/best-practices-for-api-versioning
https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/
change main.py to:
from .version1 import *
from .version2 import *
and version1.py is:
VERSION="v1"
@app.get("/")
@app.get(os.path.join(VERSION, "/index"))
async def index(request: Request):
Then refgenconf says VERSION="v1" and can be updated as needed.
we need a script that will take a refgenie.yaml config file and build the individual tar archives the server users.
this is a separate command-line tool that comes with the server package.
According to readme, the archive function only requires a config file:
"refgenieserver archive -c CONFIG
It just requires a -c argument or $REFGENIE environment variable."
However, it also appears to require genomes_desc:
refgenieserver archive -c test_multiple_servers.yaml
Traceback (most recent call last):
File "/home/user/.local/bin/refgenieserver", line 10, in <module>
sys.exit(main())
File "/home/user/.local/lib/python3.7/site-packages/refgenieserver/main.py", line 32, in main
archive(rgc, arp, args.force, args.remove, selected_cfg)
TypeError: archive() missing 1 required positional argument: 'genomes_desc'
right now we say refgenie pull -c CONFIG but it's refgenieserver -c CONFIG serve. We should make the config come after the command as it does with refgenie. I find that more intuitive.
When using --asset and/or --genome options the archiver should use the config file from genome_archive to prevent overwriting the assets metadata stored in the existing one.
Hi guys,
Thanks for maintaining such an awesome resource.
I am from the salmon developer's group and we recently released a new version of salmon that supports highly accurate quantification (check this preprint for more details). Some of our users are interested in pre built salmon indices COMBINE-lab/salmon#444 and we are exploring the resources to upload new Genome based salmon indices which are downloadable and queryable, may be based on species or the reference sequence. I found the refgenie very useful and am wondering are you guys open to updating your database with the latest salmon indices ? For starters support for just the human and mouse indices would be great too.
Not sure if this is related to the size of the asset, but refgenie pull consistently is unable to retrieve the complete file.
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 16%|████████████████████████▉ | 600M/3.75G [00:19<01:44, 30.2MB/s]
<urlopen error retrieval incomplete: got only 600489984 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 5%|███████▊ | 186M/3.75G [00:10<03:22, 17.6MB/s]
<urlopen error retrieval incomplete: got only 186474496 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 58%|██████████████████████████████████████████████████████████████████████████████████████████▏ | 2.18G/3.75G [01:06<00:47, 32.8MB/s]
<urlopen error retrieval incomplete: got only 2181976064 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 14%|█████████████████████ | 507M/3.75G [00:27<02:57, 18.2MB/s]
<urlopen error retrieval incomplete: got only 506863616 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 14%|█████████████████████▌ | 517M/3.75G [00:28<02:58, 18.2MB/s]
<urlopen error retrieval incomplete: got only 517259264 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
$ refgenie pull -g hg38 -a bowtie2
Starting pull for 'hg38/bowtie2'
'hg38/bowtie2' archive size: 3.5GB
Downloading URL: http://refgenomes.databio.org/asset/hg38/bowtie2/archive
hg38/bowtie2: 26%|█████████████████████████████████████████▏ | 990M/3.75G [00:31<01:28, 31.0MB/s]
<urlopen error retrieval incomplete: got only 989806592 out of 3749238990 bytes>
'hg38/bowtie2' download incomplete
in support of: refgenie/refgenconf#64
see failed tests: https://travis-ci.org/databio/refgenconf/jobs/595340325
@nmagee can we make the staging server operate on a different genome config file from the main server?
new entries (asset with its attributes) should be added to the config every time an asset is processed, rather than after the archive run
wait for refgenie/refgenie#26
we'll need a way for the server to get the configuration filepath it should use. perhaps we should use what we did for caravel to do this... but I can't reconcile that with how to easily run it in a container right now...
in our directions we use -v to mount the /app dir.
I know you used to need to do this, but with the switch to importing uvicorn to run on the command line, is this still necessary? can we updated the docs if not?
once refgenieconf has list genomes by asset function, refgenie/refgenconf#7
there should be an endpoint to access that.
http://refgenomes.databio.org/ is down.
It was actually down (giving a 404 error) before the release... but anyway, now I just released it and it's still down...
add to assets list:
add to index landing page:
/docs swagger auto-docsThere's a config file in this project's root that differs in structure from the one in, e.g., refgenconf's root. The one here should move to refgenconf, and then the one here should point to that one.
requirements are not included in the package, because MANIFEST.in is missing.
> pip3.6 install --user refgenieserver
Collecting refgenieserver
Using cached https://files.pythonhosted.org/packages/c3/3e/c013539317e35d22a5201129ddb21df0394e45cf079fdf5a017108fe73c0/refgenieserver-0.3.2.tar.gz
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-install-qq9uihwz/refgenieserver/setup.py'"'"'; __file__='"'"'/private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-install-qq9uihwz/refgenieserver/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-install-qq9uihwz/refgenieserver/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-install-qq9uihwz/refgenieserver/setup.py", line 13, in <module>
with open("requirements/requirements-all.txt", 'r') as reqs_file:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements/requirements-all.txt'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.```
should we make refgenie server a python package?
the swagger autodocs should include examples. right now it says 'none available'.
we just need to add examples in the code in the appropriate way to give users example queries they can try out.
see -C or --strip-components option
add the following logic to the asset pull request:
if no tag: return default (if it exists)
if tag: return that tag (if it exists)
I am unable to figure out how the refgenie archive command works. Do I give it a config file?
a few updates:
template.html, which is confusingnow the data that can be served/downloaded is represented as a list with badges providing additional info
it should be represented in a form of a set of tables, one for each genome, where assets are rows and columns are the assets' attributes
Per request of @nsheff added a dev branch to the project, based on 045ae81. Deployments using .travis.yml have been updated as well for automation. Build/deploys are taking ~60 seconds. Follow Travis-CI here
DEV/Development Branch
MASTER/Production Branch
Merging Upstream:
To Do:
dev.refgenomes.databio.org point as CNAME to dev1.uvasomrc.io.we originally described the format as just one layer deep: refgenie/refgenie#6
like:
# Genome configuration
genome_folder: $GENOMES
genome_server: http://localhost
genomes:
hg38:
bowtie2: indexed_bowtie2
hisat2: indexed_hisat2
tss_annotation: TSS.bed.gz
gtf: blah.gtf
mm10:
bowtie2: indexed_bowtie2
blacklist: blacklist/mm10.bed
rCRSd:
bowtie2: indexed_bowtie2
but the server requires a bit more information: the archive folder at the project level, and then for individual assets we need filesize, checksum, etc. So for server we need a bit more info. I therefore propose this format:
# Genome configuration
genome_folder: $GENOMES
genome_server: http://localhost
genome_archive: /path/to/archives
genomes:
hg38:
bowtie2:
path: indexed_bowtie2
archive_checksum: mm20349234n20349280345mv2035
asset_size: 32G
archive_size: 7G
hisat2:
path: indexed_hisat2
archive_checksum: mm20349234n20349280345mv2035
asset_size: 12G
archive_size: 4G
tss_annotation:
path: TSS.bed.gz
archive_checksum: mm20349234n20349280345mv2035
asset_size: 21M
archive_size: 3M
In light of #7, we need to rethink how our endpoints will share this new information. I propose:
/{genome}/{asset} - provides a complete JSON with all attributes for that asset (so, path, archive_size, asset_size, and checksum...and anything else we add).
/{genome}/{asset}/archive - downloads the archive itself (currently what we're serving with /{genome}/{asset}
I think that suffices. I don't really seed the need to allow accessing each attribute individually. The CLI can use the main JSON endpoint to get info about file sizes and checksums for display and verification (which is not possible with our current endpoints).
refgenieserver archive
email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
Traceback (most recent call last):
File "/home/nsheff/.local/bin/refgenieserver", line 10, in <module>
sys.exit(main())
File "/home/nsheff/.local/lib/python3.7/site-packages/refgenieserver/main.py", line 124, in main
archive(rgc, args)
File "/home/nsheff/.local/lib/python3.7/site-packages/refgenieserver/server_builder.py", line 26, in archive
server_rgc_path = os.path.join(rgc[CFG_ARCHIVE_KEY], os.path.basename(args.config))
File "/home/nsheff/miniconda3/lib/python3.7/posixpath.py", line 146, in basename
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
it should be reading the config from REFGENIE? or at least giving an understandable error.
right Related to #36 -- right now there is no endpoint to retrieve metadata about a genome. we will need this to implement collection-level checksums.
we need to systematically construct a list of relevant endpoints in asset splash page.
We should source them from the openAPI description JSON file by fetching just the operation_ids that describe the endpoints of interest. To do that, the operation_ids should to be stored in a dictionary in refgenconf that defines the hierarchy we need.
It appears that the ensembl_gtf asset for hg19 is either the wrong initial file or something was changed to it along the way. It should contain the following identifiers in column 3:
gzip -dc ensembl_gtf/hg19.gtf.gz | awk '{print $3}' | sort -u
CDS
exon
five_prime_utr
gene
Selenocysteine
start_codon
stop_codon
three_prime_utr
transcript
But instead it has:
CDS
exon
gene
Selenocysteine
start_codon
stop_codon
transcript
UTR
Checking the original source file, as we reference in the docs, includes the "correct" values. Not sure what the original file from http://refgenomes.databio.org is derived from...
raised by @nmagee
The FS used by the container is read-only, so we can't update the config file in order to store download counts:
INFO: ('10.0.3.50', 36128) - "GET /asset/ERCC92/bowtie1/archive HTTP/1.1" 500
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 368, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 133, in __call__
await self.error_middleware(scope, receive, send)
File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 172, in __call__
raise exc from None
File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 150, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 73, in __call__
raise exc from None
File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 62, in __call__
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 585, in __call__
await route(scope, receive, send)
File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 207, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 40, in app
response = await func(request)
File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 117, in app
raw_response = await dependant.call(**values)
File "/app/refgenieserver/main.py", line 67, in download_asset
update_stats(rgc, genome, asset)
File "/app/refgenieserver/helpers.py", line 96, in update_stats
rgc.write()
File "/usr/local/lib/python3.7/site-packages/yacman/yacman.py", line 34, in write
with open(filename, 'w') as f:
OSError: [Errno 30] Read-only file system: '/genomes/genomes.yaml'
We need an alternate way to track downloads:
Other suggestions if you need alternate ways to track counts – you could +1 to a specific key in Redis (there is a ‘counter’ data type made for that), which we have available, or to a more robust DB either relational or nosql.
revert f486b1d
implement new approach
the builder should produce and save an md5 checksum of the archive after building.
set the asset name column width to a constant value, so that the table width is equal among assets
similar to #39, but for refgenieserver archive
@nsheff can you create a square-shaped logo (maybe it's just the oil lamp from the original one) so that we can create a favicon out of it?
if you visit 'http://refgenomes.uvadcos.io/' it says: Welcome to refgenomes.databio.org.
that must be hard-coded...should probably make that banner dynamic?
is there a reason the digest endpoints return the digests with quotes?
http://staging.refgenomes.databio.org/v2/asset/hg19/fasta/default/archive_digest?tag=default
"06e47db59dad7054c4b45df7abbaef20"
Raised in conversation with @johanneskoester.
When browsing available assets via web interface, one of the things you really want to know is: what files are available in this asset (aka, which seek keys are there?). This is buried a bit, you have to click the 'attributes' tag and then look for them in the resulting json.
Should we move the seek keys to be displayed prominently on the web page?
Another possibility: could we have a "splash page" for each asset that shows all of this information more nicely formatted? It should show the seek keys, the parents, children, with links to them, for example.
This would not be hard to do; we just need a new template for a "asset detail" page that displays all info for just 1 asset, and then we need an API point that serves that for a given asset, then link to each "asset detail" page from the main asset list page. Would this solve your request, @johanneskoester?
currently there's no way to programmatically remove archives.
refgeniserver archive just checks for ones that are present in genome config but not in server genome config and builds them. And not the other way around followed by archive removal.
$ refgenieserver
Traceback (most recent call last):
File "/home/user/.local/bin/refgenieserver", line 10, in <module>
sys.exit(main())
File "/home/user/.local/lib/python3.7/site-packages/refgenieserver/main.py", line 22, in main
logger_args = dict(name=PKG_NAME, fmt=LOG_FORMAT, level=5) if args.debug else dict(name=PKG_NAME, fmt=LOG_FORMAT)
AttributeError: 'Namespace' object has no attribute 'debug'
Ideally should default to the --help messaging yes?
Related to #31
the /genome/{genome} asset says:
Returns a tarball with all the archived assets available for the genome. Requires the genome name as an input.
but the query: http://refgenomes.databio.org/genome/hg38
yeilds
{"detail":"No such genome on server"}
which is false.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.