[2025-01-21 06:25:01,050 E 23840 23933] (raylet) agent_manager.cc:83: The raylet exited immediately because one Ray agent failed, agent_name = dashboard_agent/424238335.
The raylet fate shares with the agent. This can happen because
- The version of `grpcio` doesn't follow Ray's requirement. Agent can segfault with the incorrect `grpcio` version. Check the grpcio version `pip freeze | grep grpcio`.
- The agent failed to start because of unexpected error or port conflict. Read the log `cat /tmp/ray/session_latest/logs/{dashboard_agent|runtime_env_agent}.log`. You can find the log file structure here https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#logging-directory-structure.
- The agent is killed by the OS (e.g., out of memory).
[2025-01-21 06:25:07,535 E 25705 25705] gcs_rpc_client.h:179: Failed to connect to GCS at address 192.168.0.2:46413 within 5 seconds.
[2025-01-21 06:25:07,536 E 25704 25704] gcs_rpc_client.h:179: Failed to connect to GCS at address 192.168.0.2:46413 within 5 seconds.
[2025-01-21 06:26:02,540 E 25705 25712] gcs_rpc_client.h:664: Failed to connect to GCS within 60 seconds. GCS may have been killed. It's either GCS is terminated by `ray stop` or is killed unexpectedly. If it is killed unexpectedly, see the log file gcs_server.out. https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#logging-directory-structure. The program will terminate.
[2025-01-21 06:26:02,541 E 25704 25731] gcs_rpc_client.h:664: Failed to connect to GCS within 60 seconds. GCS may have been killed. It's either GCS is terminated by `ray stop` or is killed unexpectedly. If it is killed unexpectedly, see the log file gcs_server.out. https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#logging-directory-structure. The program will terminate.