| [2025-01-15 22:09:24,293 C 14019 14019] (raylet) node_manager.cc:1043: [Timeout] Exiting because this node manager has mistakenly been marked as dead by the GCS: GCS failed to check the health of this node for 5 times. This is likely because the machine or raylet has become overloaded. | |
| *** StackTrace Information *** | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0xbdf73a) [0x55f20d06173a] ray::operator<<() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0xbe1b21) [0x55f20d063b21] ray::RayLog::~RayLog() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x323299) [0x55f20c7a5299] ray::raylet::NodeManager::NodeRemoved() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x536e69) [0x55f20c9b8e69] ray::gcs::NodeInfoAccessor::HandleNotification() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x669e98) [0x55f20caebe98] EventTracker::RecordExecution() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x664e8e) [0x55f20cae6e8e] std::_Function_handler<>::_M_invoke() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x665306) [0x55f20cae7306] boost::asio::detail::completion_handler<>::do_complete() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0xc53f9b) [0x55f20d0d5f9b] boost::asio::detail::scheduler::do_run_one() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0xc56529) [0x55f20d0d8529] boost::asio::detail::scheduler::run() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0xc56a42) [0x55f20d0d8a42] boost::asio::io_context::run() | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x1e9155) [0x55f20c66b155] main | |
| /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3cf7e48d90] | |
| /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f3cf7e48e40] __libc_start_main | |
| /usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet(+0x243277) [0x55f20c6c5277] | |