Comments (3)
Another useful feature:
[ ] a wrapper that handles the IOErrors that we sometimes get with large worker numbers and keeps submitting the function until it succeeds or reaches a Max retry num
I feel like some of this stuff could be developed on dask_kubernetes but maybe easier to get it up and running here and then see if it can be merged
from rhg_compute_tools.
yeah. this is just like a helper function that handles errors that frequently pop up in the chaos of crapton-of-workers land and then hammers the jobs until they complete?
I've actually found the cluster to be much more stable, even when running huge numbers of jobs. Have you encountered this recently?
from rhg_compute_tools.
nice! I haven't run a huge number of jobs in a long time (like since BR1 push). But yeah that's what I was thinking. I remember the IOError being the main issue. If we start experiencing this again, we can try to build in something like that maybe
from rhg_compute_tools.
Related Issues (20)
- gcs.cp does not work for sources not mounted on fuse
- fix gcs.rm
- add globals blocker HOT 3
- Add a "make gcs fuse directories in place" function HOT 1
- Setting environment variables on remote cluster does not work
- CI broken: pytest raises "fixture applied more than once"
- Inconsistency in API for different gsutil commpands
- Couple of bugs in kubernetes.py HOT 2
- enforce isort and black in tests HOT 2
- switch from bumpversion to setuptools_scm HOT 2
- getting clusters is confusing af
- Drop pytest-runner use
- rhg_compute_tools.xarray.dataarrays_from_delayed can blow up memory
- add alternative combining methods to rct.xarray.*_from_delayed HOT 2
- rhg_compute_tools.gcs.cp might not always do what we expect HOT 2
- Should we instantiate a k8s cluster within CI for rhg_compute_tools.kubernetes unit tests? HOT 3
- efficient-geopandas-nearest-neighbor HOT 5
- mpl 3.4 drops `font_manager._rebuild()` function and breaks `design` imports HOT 1
- Figure out how to use RHG fonts/styles HOT 2
- extra conda packages kwarg fails with dask gateway HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rhg_compute_tools.