Comments (6)
Another option: unpack the imagecodecs wheel, remove unneeded *.so files, repack the wheel, and install it. See https://wheel.readthedocs.io/en/stable/reference/wheel_pack.html#examples
from tiffslide.
hmmm, good question...
...
888K ./urllib3
1,1M ./pkg_resources
1,2M ./tifffile
1,4M ./yarl
1,4M ./zarr
1,6M ./chardet
2,8M ./PIL
3,8M ./setuptools
7,5M ./aiohttp
7,7M ./Pillow.libs
11M ./pip
26M ./numcodecs
30M ./numpy
33M ./numpy.libs
38M ./imagecodecs.libs
60M ./botocore
66M ./imagecodecs
295M .
It seems like imagecodecs is the biggest offender with ~104MB total. And when you install s3fs, botocore is another 60MB.
I'll have to think about it a bit. Basically all these dependencies are pulled in via tifffile.
from tiffslide.
Hi @rmontroy
So the way to go seems to be to install only what you require to decode the svs images you want to process.
imagecodecs
can be built with a subset of supported formats by skipping everything else:
For example on Ubuntu 20.04:
### Disabled by default
# --global-option="--skip-avif" \
# --global-option="--skip-brunsli" \
# --global-option="--skip-jpegls" \
# --global-option="--skip-jpegxl" \
# --global-option="--skip-lerc" \
# --global-option="--skip-lz4f" \
# --global-option="--skip-zfp" \
# --global-option="--skip-zlibng" \
### Required for the tiffslide tests to pass
# --global-option="--skip-shared" \
# --global-option="--skip-imcd" \
# --global-option="--skip-jpeg8" \
python -m pip install imagecodecs \
--global-option="build_ext" \
--global-option="--skip-aec" \
--global-option="--skip-bitshuffle" \
--global-option="--skip-blosc" \
--global-option="--skip-brotli" \
--global-option="--skip-bz2" \
--global-option="--skip-deflate" \
--global-option="--skip-gif" \
--global-option="--skip-jpeg2k" \
--global-option="--skip-jpegxr" \
--global-option="--skip-lz4" \
--global-option="--skip-lzf" \
--global-option="--skip-lzma" \
--global-option="--skip-pglz" \
--global-option="--skip-png" \
--global-option="--skip-rcomp" \
--global-option="--skip-snappy" \
--global-option="--skip-tiff" \
--global-option="--skip-webp" \
--global-option="--skip-zlib" \
--global-option="--skip-zopfli" \
--global-option="--skip-zstd"
The above command installs a version imagecodecs that only has the required formats to make the tiffslide tests pass. It could be that when you test with an svs of yours, an error will be raised and you need to rebuild imagecodecs and leave out the specific skip option to prevent that error for your file (i.e. maybe jpeg2k or so...).
Here is a link to the relevant instructions in imagecodecs: https://github.com/cgohlke/imagecodecs/blob/e92cef6c1878f0b69ebbfb33f9bb809eccbdc31e/imagecodecs/imagecodecs.py#L160-L181 You might have to install the system build dependencies if the build fails for you.
I hope that helps. Let me know how things go.
Cheers,
Andreas 😃
PS.: It could be that we would need to build a manylinux wheel to make it work since we're linking against os libraries.
PSS.: Another option could be to manually remove the unneeded *.so files in the site-packages/imagecodecs folder before you create you lambda layer.
from tiffslide.
@ap-- Any interest in creating a public AWS Lambda layer that's as small as possible? I don't have the time to look into it now, but I'd use it if it already existed, provided it performed well enough.
from tiffslide.
I'll have a look. I might have some time next week to make the layer or at least an easy way to create a minimal venv.
from tiffslide.
FYI, there is a straightforward solution, using the serverless framework and this plugin:
https://www.serverless.com/plugins/serverless-python-requirements
Key configuration is to add:
custom:
pythonRequirements:
zip: true
And to include this in your handler:
try:
import unzip_requirements
except ImportError:
pass
This zips all the requirements (including tiffslide) so that it falls below the cutoff by a wide margin. This does add a second or two to cold starts, but there is no cost for warm starts.
from tiffslide.
Related Issues (20)
- [Bug] Padding is expected. But it is not padding. HOT 1
- CI: cache downloads
- Support for axes=='YX' and uint16 svs files. HOT 5
- Issue with `get_best_level_for_downsample` for NDPI HOT 2
- Compatibility issue with read_region HOT 5
- test_from_kerchunk fails in CI but passes locally? HOT 2
- tiffslide with CLAM HOT 5
- Offer direct-to-gpu decode HOT 1
- round downsamples ?!?
- tiffslide fails to detect svs format and MPP if ImageDescription contains non ascii characters HOT 11
- Refactor leica metadata parsing
- Support for Hamamatsu VMS
- Cannot parse metadata of tiled-tiff images. HOT 4
- Tiffslide much slower than openslide reading patches from SVS with JPEG2000 compression HOT 31
- Dependencies: restrict combinations of versions of dependencies
- tiffslide not reading MPP of CAMELYON16 but openslide does HOT 3
- integration with xarray? HOT 11
- Support icc profile HOT 1
- quickhash implementation
- garbled tiles from CAMELYON16 dataset using tiffslide but not openslide HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tiffslide.