Comments (5)
I have heard of this technique referred to as "bring your own buffer"...It may be possible to do this. Do you have significant evidence of the performance degradation?
I can see you are using scale: 1 so that would probably be intensive across the whole length of a chromosome. You could consider using one of the other reductionLevels to make this involve less data, probably would be faster, but if you require the lowest scale then I can see that would be resource intensive.
from bbi-js.
Yes, we indeed require to use scale: 1
. We're converting bigWig files into loompy files for an implementation of the ga4gh-rnaseq API, and we need to fill a buffer with every value, and this is for multiple tracks at once so we're filling lots of buffers with lots of entries. The API also allows for returning multiple tracks of the whole bigWig file, I'm pretty sure that any saved allocation can decrease the memory-cost of the process. (speed is not an issue though, I meant performance in terms of memory)
from bbi-js.
I'll just ask a couple more questions
- Is there anything particular about this library (bbi-js) that makes this particularly well suited to your app? Do you do these conversions on the fly? I am imagining for a large data ingestion, I would probably just convert to bedgraph or regular wig and stream that into your data warehouse/hdf5/loompy
- Do you have interest in implementing this yourself? I had another request similar to this here GMOD/generic-filehandle#20 and I'd love to see progress on it but until it becomes a bottleneck for my use cases (primarily genome browser apps) it's hard for me to push it into the priority queue
from bbi-js.
The purpose is compatibility with https://github.com/romgrk/node-loompy ? so it keeps it all in the js ecosystem?
from bbi-js.
Yes, we're using that module that we wrote to keep it all in JS.
For your points:
- We have tons (not sure how many but >10,000, maybe >100,000) of bigWig tracks and those files are provided to us in that format and we need them that way for other purposes, so it's not practical to convert them and have them in both formats as we would run out of space.
- Sure, I'll try to find some time and open a PR
from bbi-js.
Related Issues (20)
- Unhandled Rejection (TypeError): fsOpen is not a function HOT 8
- Guaranteed order of returned results
- Proposal: Use async iterator instead of observable
- Example doc? HOT 10
- pre-compile parser objects for performance HOT 4
- Version parsers HOT 3
- Allow accessing zoom level directly instead of via scale/bpPerScale HOT 1
- Method to enumerate all features in BigBed file
- Use generator to emit data as it is received HOT 1
- Aborting not being caught? HOT 1
- RangeError: Trying to access beyond buffer length HOT 4
- Parse/test names index in bigbed HOT 1
- Possible alternative compression support
- ESM export is commonjs HOT 8
- Possible alternative format support
- Improve caching behavior HOT 2
- Error while parsing a custom-format bigbed file HOT 4
- TypeError: this.ranges.map is not a function HOT 6
- need more doc in readme HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bbi-js.