Comments (6)
This is a great point. The assumption is that all the data from the task doesn't fit in the object store, right? In situations where we have plenty of memory and reconstruction is only being triggered due to node failures, this example should work, right?
In the case where things don't fit in memory, it'll be tough to handle this example correctly. Spilling over to disk would extend the range of situations that get handled appropriately, but we could come up with workloads where that wouldn't save us.
Btw, I think the same problem is present even without the initial ray.put
. if you just did arg = array
, then the first time you called single_dependency.remote(arg)
, ray.put
would be called under the hood on the argument (this happens for basically all large or complicated arguments).
from ray.
Are there proposals for solving this issue? Have you considered adding tracing to the object store to detect repeated evictions and return an error? Have you considered adding user-controlled pinning to prevent the object store from evicting data?
from ray.
We haven't seen this problem occur in practice, so we haven't prioritized it a huge amount.
I'd eventually like to have ways to give hints to the object store about which objects won't be needed anymore and which ones will. For object's that contain numpy arrays, it's already sort of possible to prevent an object from being evicted by calling x = ray.get(x_id)
on the object ID. As long as x
is in scope, the object won't be evicted from the local object store (because the numpy arrays are backed by shared memory in the object store). However, I wouldn't want people to rely on that implementation detail.
Detecting repeated evictions and acting based on that is definitely doable. It'd require some thought about which component would be responsible for doing that.
As for concrete proposals, we'd like to be able to diagnose precisely when this is happening and make that information available to the user (this may be doable with a count of how many times each object has been reconstructed).
from ray.
Here is a solution that @pcmoritz and I discussed that we think will cover this case as long as a task's ray.put
objects fit in the object store.
The key issue is that by having a task do a ray.put
, and then calling ray.get
on other tasks that are dependent on the put value, we create a cyclic dependency. The caller waits on the return values of the other tasks, but the other tasks wait on the object that was put by the caller. If we lose that object but the caller is still waiting on the other tasks in a ray.get
, then we will deadlock because we can't re-execute the caller.
We can avoid deadlock by ensuring that the object is available when the other tasks execute. We can do this by having the caller pin the object to its local object store. It's okay if that node dies, because then we will lose both the object and the caller, so we can simply re-execute the caller to reconstruct the object. If it's an option, we can also "pin" the object by replicating it to another node or writing it to disk, in case there's not enough memory in the local store.
Another option that is more radical is to break the cycle completely by splitting the caller into two tasks, one which returns the values that it called ray.put
on and another that is the continuation. This would require us to introspect and mutate application code, and it's unlikely that we'll cover all cases in Python.
from ray.
The first solution (pinning) makes sense. In that case we should consider enhancing the plasma API to support this kind of thing so we don't have to keep creating dummy numpy arrays.
from ray.
Pinning the puts in the object store of the local node makes sense to me. Implementation wise, I assume this means that when a task puts an object, we record this so that we remember to add the object entry to the gcs when the task finishes and then unpin the object.
from ray.
Related Issues (20)
- [Core | Ray on Spark] Allow cluster_mode='yarn' for Ray on Spark
- [Core] Raylet may crash when the job exits HOT 2
- CI test windows://python/ray/tests:test_network_failure_e2e is consistently_failing HOT 2
- [Ray Serve] Running experimental multiple application in different containers on EKS HOT 3
- [core][accelerated DAGs] Assertion check fails when driver exits during teardown
- Serve: Add support for Podman 5.x for multi-container applications
- [Core] shall we document `__ray_call__` as a public API?
- Serialization Issue HOT 12
- CI test linux://rllib:TestLearnerGroupAsyncUpdate is consistently_failing HOT 4
- CI test linux://rllib:learning_tests_carpole_dqn_envrunner is flaky HOT 7
- [Serve] Exited containers not cleaned up HOT 3
- [Air] ray.air.callbacks.wandb still exists, but fails on import due to `warn_structure_refactor` no longer existing
- [Core] Temporary folder per worker process HOT 2
- [RLlib] ray.tune.error.TuneError: Stopping criteria num_env_steps_sampled_lifetime not provided in result dict.
- [data] autodoc mishandling type annotations HOT 1
- [Rllib] Rllib provides wrong state batch size during "bug check" batches on torch custom model
- Release test chaos_torch_batch_inference_16_gpu_300gb_raw.aws failed HOT 2
- [RLlib] PPO does not respect `sample_timeout_s` with old and hybrid API stacks
- Ray Cluster: Failed to create a ray cluster using running container
- [RLlib|Tune] - Video results do not log correctly in wandb
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ray.