[2025-01-14 17:59:27,854 I 80547 80547] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed [2025-01-14 17:59:27,855 I 80547 80547] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. [2025-01-14 17:59:27,862 I 80547 80547] (gcs_server) event.cc:493: Ray Event initialized for GCS [2025-01-14 17:59:27,862 I 80547 80547] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE [2025-01-14 17:59:27,862 I 80547 80547] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR [2025-01-14 17:59:27,862 I 80547 80547] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB [2025-01-14 17:59:27,862 I 80547 80547] (gcs_server) event.cc:324: Set ray event level to warning [2025-01-14 17:59:27,868 I 80547 80547] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:42: Loading job table data. [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:54: Loading node table data. [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:80: Loading actor table data. [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 [2025-01-14 17:59:27,870 I 80547 80547] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 621036b73cc6dc08763045727f24862a937210c207488a0c5b0124b6 [2025-01-14 17:59:27,871 I 80547 80547] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 [2025-01-14 17:59:27,873 I 80547 80547] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 64748. [2025-01-14 17:59:28,117 I 80547 80547] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 0 - DrainNode request count: 0 - GetAllNodeInfo request count: 0 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 0 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-14 17:59:28,117 I 80547 80547] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 25 total (5 active) Queueing time: mean = 87.947 ms, max = 243.137 ms, min = 2.023 us, total = 2.199 s Execution time: mean = 9.858 ms, total = 246.455 ms Event stats: GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.040 ms, total = 243.359 ms, Queueing time: mean = 188.464 ms, max = 242.552 ms, min = 2.023 us, total = 1.696 s GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 7.726 us, total = 38.632 us, Queueing time: mean = 52.164 us, max = 61.699 us, min = 45.507 us, total = 260.821 us PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 3.985 us, total = 15.940 us, Queueing time: mean = 121.526 ms, max = 243.137 ms, min = 242.965 ms, total = 486.102 ms event_loop_lag_probe - 2 total (0 active), Execution time: mean = 7.256 us, total = 14.512 us, Queueing time: mean = 7.439 ms, max = 14.389 ms, min = 489.973 us, total = 14.879 ms NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 3.006 ms, total = 3.006 ms, Queueing time: mean = 1.249 ms, max = 1.249 ms, min = 1.249 ms, total = 1.249 ms RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 20.949 us, total = 20.949 us, Queueing time: mean = 2.940 us, max = 2.940 us, min = 2.940 us, total = 2.940 us [2025-01-14 17:59:28,117 I 80547 80547] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 5 total (1 active) Queueing time: mean = 93.799 us, max = 329.628 us, min = 12.221 us, total = 468.994 us Execution time: mean = 75.762 us, total = 378.809 us Event stats: event_loop_lag_probe - 3 total (0 active), Execution time: mean = 122.594 us, total = 367.782 us, Queueing time: mean = 126.873 us, max = 329.628 us, min = 12.221 us, total = 380.618 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.027 us, total = 11.027 us, Queueing time: mean = 88.376 us, max = 88.376 us, min = 88.376 us, total = 88.376 us GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-14 17:59:28,118 I 80547 80547] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4 total (1 active) Queueing time: mean = 581.912 us, max = 1.151 ms, min = 92.823 us, total = 2.328 ms Execution time: mean = 1.167 ms, total = 4.667 ms Event stats: event_loop_lag_probe - 2 total (0 active), Execution time: mean = 2.328 ms, total = 4.656 ms, Queueing time: mean = 1.117 ms, max = 1.151 ms, min = 1.084 ms, total = 2.235 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.700 us, total = 11.700 us, Queueing time: mean = 92.823 us, max = 92.823 us, min = 92.823 us, total = 92.823 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-14 17:59:28,118 I 80547 80547] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4 total (0 active) Queueing time: mean = 2.091 ms, max = 7.789 ms, min = 42.847 us, total = 8.365 ms Execution time: mean = 54.949 us, total = 219.797 us Event stats: RaySyncerRegister - 2 total (0 active), Execution time: mean = 581.000 ns, total = 1.162 us, Queueing time: mean = 266.853 us, max = 267.639 us, min = 266.068 us, total = 533.707 us event_loop_lag_probe - 2 total (0 active), Execution time: mean = 109.317 us, total = 218.635 us, Queueing time: mean = 3.916 ms, max = 7.789 ms, min = 42.847 us, total = 7.832 ms [2025-01-14 17:59:30,401 I 80547 80547] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:30,401 I 80547 80547] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:30,401 I 80547 80547] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 registered, will try to reschedule all the infeasible placement groups. [2025-01-14 17:59:30,405 I 80547 80625] (gcs_server) ray_syncer.cc:377: Get connection node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:31,232 I 80547 80547] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 80479 [2025-01-14 17:59:31,232 I 80547 80547] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 80479 [2025-01-14 17:59:33,209 I 80547 80547] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000 [2025-01-14 17:59:33,252 I 80547 80547] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,252 I 80547 80547] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,253 I 80547 80547] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,253 I 80547 80547] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,254 I 80547 80547] (gcs_server) gcs_resource_manager.cc:161: [UpdateFromResourceView]: received resource usage from unknown node id 7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,403 I 80547 80596] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,403 I 80547 80596] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=7cb9ec43b63e071a2e98aa563fcaad934d117812ad3006115007f556 [2025-01-14 17:59:33,415 I 80547 80547] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down... [2025-01-14 17:59:33,417 I 80547 80547] (gcs_server) gcs_server.cc:267: Stopping GCS server. [2025-01-14 17:59:33,558 I 80547 80547] (gcs_server) gcs_server.cc:284: GCS server stopped. [2025-01-14 17:59:33,559 I 80547 80547] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped. [2025-01-14 17:59:33,568 I 80547 80547] (gcs_server) stats.h:120: Stats module has shutdown.