Comments (2)
ruisearch42
commented on May 19, 2024
There was a segfault when importing pyarrow:
platform darwin -- Python 3.9.18, pytest-7.0.1, pluggy-1.3.0 -- /usr/local/opt/miniconda/bin/python3
cachedir: .pytest_cache
rootdir: /Users/ec2-user/.buildkite-agent/builds/bk-macos-branch-queue-i-0a7b36e54bbaab7d5-1/ray-project/postmerge-macos, configfile: pytest.ini
plugins: asyncio-0.16.0, docker-tools-3.1.3, sphinx-0.5.1.dev0, forked-1.4.0, sugar-0.9.5, timeout-2.1.0, shutil-1.7.0, rerunfailures-11.1.2, virtualenv-1.7.0, anyio-3.7.1, lazy-fixture-0.6.3, httpserver-1.0.6
timeout: 180.0s
timeout method: signal
timeout func_only: False
collecting ... collected 1 item
python/ray/tests/test_actor_failures.py::test_exit_actor_queued 2024-03-25 20:03:33,628 WARNING worker.py:1503 -- File descriptor limit 256 is too low for production servers and may result in connection errors. At least 8192 is recommended. --- Fix with 'ulimit -n 8192'
2024-03-25 20:03:35,840 INFO worker.py:1743 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8266
(RegressionAsync pid=79092) *** SIGSEGV received at time=1711397017 ***
(RegressionAsync pid=79092) PC: @ 0x10d48a277 (unknown) PyObject_RichCompareBool
(RegressionAsync pid=79092) [2024-03-25 20:03:37,049 E 79092 1111773] logging.cc:361: *** SIGSEGV received at time=1711397017 ***
(RegressionAsync pid=79092) [2024-03-25 20:03:37,049 E 79092 1111773] logging.cc:361: PC: @ 0x10d48a277 (unknown) PyObject_RichCompareBool
(RegressionAsync pid=79092) Fatal Python error: Segmentation fault
(RegressionAsync pid=79092)
(RegressionAsync pid=79092) Stack (most recent call first):
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1554 in find_spec
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1395 in _get_spec
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1423 in find_spec
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 925 in _find_spec
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 982 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1181 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1181 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "/usr/local/opt/miniconda/lib/python3.9/site-packages/numpy/random/_pickle.py", line 1 in <module>
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 850 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
(RegressionAsync pid=79092) File "/usr/local/opt/miniconda/lib/python3.9/site-packages/numpy/random/__init__.py", line 180 in <module>
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 850 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
(RegressionAsync pid=79092) File "/usr/local/opt/miniconda/lib/python3.9/site-packages/numpy/__init__.py", line 152 in <module>
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 850 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 1181 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "/usr/local/opt/miniconda/lib/python3.9/site-packages/pyarrow/__init__.py", line 65 in <module>
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap_external>", line 850 in exec_module
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
(RegressionAsync pid=79092) File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
(RegressionAsync pid=79092) File "/Users/ec2-user/.buildkite-agent/builds/bk-macos-branch-queue-i-0a7b36e54bbaab7d5-1/ray-project/postmerge-macos/python/ray/_private/arrow_serialization.py", line 46 in _register_custom_datasets_serializers
(RegressionAsync pid=79092) File "/Users/ec2-user/.buildkite-agent/builds/bk-macos-branch-queue-i-0a7b36e54bbaab7d5-1/ray-project/postmerge-macos/python/ray/util/serialization_addons.py", line 39 in apply
(RegressionAsync pid=79092) File "/Users/ec2-user/.buildkite-agent/builds/bk-macos-branch-queue-i-0a7b36e54bbaab7d5-1/ray-project/postmerge-macos/python/ray/_private/serialization.py", line 153 in __init__
(RegressionAsync pid=79092) File "/Users/ec2-user/.buildkite-agent/builds/bk-macos-branch-queue-i-0a7b36e54bbaab7d5-1/ray-project/postmerge-macos/python/ray/_private/worker.py", line 679 in get_serialization_context
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff37ffcca2d8d3ecddbf31a11b01000000 Worker ID: 2064f2064e70f96ad4380a97dd28cd28ad98366e1a8cc450ee6eeaf0 Node ID: 38f3224f8612cf76fadabffc9b3676230789c3f7a21c3ae5001237c6 Worker IP address: 127.0.0.1 Worker port: 52945 Worker PID: 79092 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(raylet) Warning: More than 5000 tasks are pending submission to actor 37ffcca2d8d3ecddbf31a11b01000000. To reduce memory usage, wait for these tasks to finish before sending more.
FAILED
from ray.
can-anyscale
commented on May 19, 2024
Test passed on latest run: https://buildkite.com/ray-project/postmerge-macos/builds/314#018e8bde-907d-44a3-8454-7b350571585c
from ray.
Related Issues (20)
- Release test rllib_learning_tests_sac_tf.aws failed HOT 2
- Release test rllib_learning_tests_impala_old_api_stack_tf.aws failed HOT 2
- Release test rllib_learning_tests_cql_old_api_stack_tf.aws failed HOT 1
- Release test rllib_learning_tests_marwil_old_api_stack_tf.aws failed HOT 1
- [RLlib] MultiAgentEnv grouping wrong example
- [Ray data] [stable diffussion batch inference] cpu resources in cluster cannot be fully utilized when running stable diffusion batch inference task.
- [Ray Core] cannot get history job log and task summaries after head node re-create HOT 1
- [Data] `write_bigquery` doesn't support properly list-typed fields
- [train+tune] How do I tune over hyperparameters for Ray Train distributed training runs? HOT 4
- platform toolchain related error in ray installing source HOT 1
- CI test linux://python/ray/data:doctest is consistently_failing HOT 3
- [release-test] `single_node_oom.aws_1712366943` failed HOT 1
- [job] Job log tailer doesn't work with binary data. HOT 5
- [Data] Ray Data doesn't account for object store memory from object dtypes HOT 2
- [Core] Pending tasks gets hanging instead of resubmitted
- [Core] Ray schedules to nodes after they are drained
- [core] Possible performance regression in many_actors nightly benchmark HOT 2
- [Core] Function to determine in-object-store size of just-yielded object
- [data] DatasetLogger not support env RAY_LOG_TO_STDERR = "1"
- [Core] Something wrong in a `ray.remote` walkthrough example
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ray.