ray/session_2025-01-10_23-12-26_972222_49857/logs/gcs_server.out

[2025-01-10 23:12:27,005 I 49938 49938] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-10 23:12:27,006 I 49938 49938] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-10 23:12:27,010 I 49938 49938] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-10 23:12:27,010 I 49938 49938] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-10 23:12:27,010 I 49938 49938] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-10 23:12:27,010 I 49938 49938] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-10 23:12:27,010 I 49938 49938] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-10 23:12:27,012 I 49938 49938] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: dc63ec7d4f2695faf284f4494bda099531bef1598f2f452a1cec6669
[2025-01-10 23:12:27,013 I 49938 49938] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-10 23:12:27,015 I 49938 49938] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 41584.
[2025-01-10 23:12:27,216 I 49938 49938] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-10 23:12:27,216 I 49938 49938] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 23 total (4 active)
Queueing time: mean = 79.640 ms, max = 203.182 ms, min = 1.777 us, total = 1.832 s
Execution time:  mean = 8.838 ms, total = 203.277 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 22.579 ms, total = 203.208 ms, Queueing time: mean = 157.578 ms, max = 202.802 ms, min = 1.777 us, total = 1.418 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 7.516 us, total = 37.579 us, Queueing time: mean = 44.588 us, max = 48.313 us, min = 41.479 us, total = 222.942 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 1.962 us, total = 7.846 us, Queueing time: mean = 101.568 ms, max = 203.182 ms, min = 203.091 ms, total = 406.272 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 5.898 us, total = 11.796 us, Queueing time: mean = 3.508 ms, max = 6.860 ms, min = 155.568 us, total = 7.016 ms
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 11.950 us, total = 11.950 us, Queueing time: mean = 2.622 us, max = 2.622 us, min = 2.622 us, total = 2.622 us
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-10 23:12:27,216 I 49938 49938] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 4 total (1 active)
Queueing time: mean = 143.371 us, max = 496.147 us, min = 14.899 us, total = 573.485 us
Execution time:  mean = 107.104 us, total = 428.417 us
Event stats:
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 210.040 us, total = 420.079 us, Queueing time: mean = 279.293 us, max = 496.147 us, min = 62.439 us, total = 558.586 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.338 us, total = 8.338 us, Queueing time: mean = 14.899 us, max = 14.899 us, min = 14.899 us, total = 14.899 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-10 23:12:27,216 I 49938 49938] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 4 total (1 active)
Queueing time: mean = 217.588 us, max = 688.962 us, min = 15.092 us, total = 870.353 us
Execution time:  mean = 96.158 us, total = 384.633 us
Event stats:
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 181.049 us, total = 362.098 us, Queueing time: mean = 427.630 us, max = 688.962 us, min = 166.299 us, total = 855.261 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 22.535 us, total = 22.535 us, Queueing time: mean = 15.092 us, max = 15.092 us, min = 15.092 us, total = 15.092 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-10 23:12:27,216 I 49938 49938] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 4 total (0 active)
Queueing time: mean = 277.287 us, max = 1.047 ms, min = 12.039 us, total = 1.109 ms
Execution time:  mean = 74.859 us, total = 299.438 us
Event stats:
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 720.500 ns, total = 1.441 us, Queueing time: mean = 12.662 us, max = 13.285 us, min = 12.039 us, total = 25.324 us
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 148.999 us, total = 297.997 us, Queueing time: mean = 541.912 us, max = 1.047 ms, min = 36.937 us, total = 1.084 ms


[2025-01-10 23:12:28,407 I 49938 49938] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:28,408 I 49938 49938] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:28,408 I 49938 49938] (gcs_server) gcs_placement_group_manager.cc:819: A new node: df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0 registered, will try to reschedule all the infeasible placement groups.
[2025-01-10 23:12:28,412 I 49938 50014] (gcs_server) ray_syncer.cc:377: Get connection node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:29,364 I 49938 49938] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 49857
[2025-01-10 23:12:29,364 I 49938 49938] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 49857
[2025-01-10 23:12:37,021 W 49938 49961] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:60285: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-10 23:12:47,134 I 49938 49938] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-10 23:12:47,239 I 49938 49938] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,239 I 49938 49938] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,239 I 49938 49938] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,239 I 49938 49938] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,348 I 49938 49987] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,348 I 49938 49987] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=df7ee78b4bc3ac4f066532a9ba0bb0c580f959d04b14b9c1859f7fa0
[2025-01-10 23:12:47,352 I 49938 49938] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-10 23:12:47,353 I 49938 49938] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-10 23:12:47,419 I 49938 49938] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-10 23:12:47,419 I 49938 49938] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-10 23:12:47,431 I 49938 49938] (gcs_server) stats.h:120: Stats module has shutdown.