ray/session_2025-01-21_05-47-33_016171_18344/logs/gcs_server.out

[2025-01-21 05:47:33,311 I 18551 18551] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-21 05:47:33,311 I 18551 18551] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-21 05:47:33,325 I 18551 18551] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: cf4503ed329bc2b5612e9c435582c6da51e2e7d5bac7b8183e0cfd01
[2025-01-21 05:47:33,327 I 18551 18551] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-21 05:47:33,330 I 18551 18551] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 56605.
[2025-01-21 05:47:33,581 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:47:33,581 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 91.127 ms, max = 251.846 ms, min = 2.247 us, total = 2.278 s
Execution time:  mean = 10.194 ms, total = 254.838 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.986 ms, total = 251.870 ms, Queueing time: mean = 195.243 ms, max = 251.330 ms, min = 2.247 us, total = 1.757 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 9.914 us, total = 49.568 us, Queueing time: mean = 74.010 us, max = 93.369 us, min = 64.515 us, total = 370.049 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 5.246 us, total = 20.983 us, Queueing time: mean = 125.891 ms, max = 251.846 ms, min = 251.719 ms, total = 503.565 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 12.339 us, total = 24.678 us, Queueing time: mean = 7.565 ms, max = 14.800 ms, min = 328.606 us, total = 15.129 ms
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 2.857 ms, total = 2.857 ms, Queueing time: mean = 1.926 ms, max = 1.926 ms, min = 1.926 ms, total = 1.926 ms
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 15.587 us, total = 15.587 us, Queueing time: mean = 3.595 us, max = 3.595 us, min = 3.595 us, total = 3.595 us


[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 1.549 ms, max = 7.604 ms, min = 9.340 us, total = 7.746 ms
Execution time:  mean = 52.263 us, total = 261.314 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 83.150 us, total = 249.449 us, Queueing time: mean = 2.551 ms, max = 7.604 ms, min = 9.340 us, total = 7.652 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 2.042 ms, max = 8.828 ms, min = 9.247 us, total = 10.210 ms
Execution time:  mean = 34.846 us, total = 174.231 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 43.227 us, total = 129.680 us, Queueing time: mean = 2.974 ms, max = 8.828 ms, min = 9.247 us, total = 8.921 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 5 total (0 active)
Queueing time: mean = 229.554 us, max = 779.592 us, min = 11.557 us, total = 1.148 ms
Execution time:  mean = 259.601 us, total = 1.298 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 432.013 us, total = 1.296 ms, Queueing time: mean = 319.245 us, max = 779.592 us, min = 11.557 us, total = 957.736 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d registered, will try to reschedule all the infeasible placement groups.
[2025-01-21 05:47:36,744 I 18551 18638] (gcs_server) ray_syncer.cc:377: Get connection node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:37,747 I 18551 18551] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 18344
[2025-01-21 05:47:37,747 I 18551 18551] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 18344
[2025-01-21 05:47:43,332 W 18551 18574] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:64843: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-21 05:48:33,582 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 12

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 13
- cumulative published bytes: 12684
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 5
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:48:33,582 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 1113 total (4 active)
Queueing time: mean = 2.096 ms, max = 251.846 ms, min = 853.000 ns, total = 2.333 s
Execution time:  mean = 529.727 us, total = 589.586 ms
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 22.461 us, total = 5.436 ms, Queueing time: mean = 71.164 us, max = 14.800 ms, min = 2.721 us, total = 17.222 ms
	GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.776 ms, total = 253.981 ms, Queueing time: mean = 12.318 ms, max = 251.330 ms, min = 2.247 us, total = 1.762 s
	GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 23.769 us, total = 1.711 ms, Queueing time: mean = 20.905 us, max = 337.749 us, min = 3.595 us, total = 1.505 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 46.636 us, total = 3.311 ms, Queueing time: mean = 103.644 us, max = 240.717 us, min = 15.205 us, total = 7.359 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 677.549 us, total = 48.106 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	RayletLoadPulled - 60 total (1 active), Execution time: mean = 242.463 us, total = 14.548 ms, Queueing time: mean = 73.646 us, max = 176.027 us, min = 16.795 us, total = 4.419 ms
	NodeManagerService.grpc_client.GetResourceLoad - 56 total (0 active), Execution time: mean = 2.205 ms, total = 123.453 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 56 total (0 active), Execution time: mean = 78.094 us, total = 4.373 ms, Queueing time: mean = 150.019 us, max = 2.124 ms, min = 16.199 us, total = 8.401 ms
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 40.728 us, total = 1.588 ms, Queueing time: mean = 112.935 us, max = 254.636 us, min = 31.997 us, total = 4.404 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 626.221 us, total = 24.423 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 35.048 us, total = 841.162 us, Queueing time: mean = 108.178 us, max = 186.281 us, min = 30.232 us, total = 2.596 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 597.630 us, total = 14.343 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 21 total (1 active), Execution time: mean = 12.154 us, total = 255.234 us, Queueing time: mean = 78.771 us, max = 151.718 us, min = 40.254 us, total = 1.654 ms
	HealthCheck - 18 total (0 active), Execution time: mean = 8.633 us, total = 155.395 us, Queueing time: mean = 105.561 us, max = 133.874 us, min = 29.147 us, total = 1.900 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 734.030 us, total = 8.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 95.296 us, total = 1.144 ms, Queueing time: mean = 132.082 us, max = 190.945 us, min = 32.307 us, total = 1.585 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.298 ms, total = 7.791 ms, Queueing time: mean = 68.888 us, max = 137.944 us, min = 11.375 us, total = 413.330 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 104.086 us, total = 312.259 us, Queueing time: mean = 122.171 us, max = 138.303 us, min = 101.321 us, total = 366.512 us
	GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 2608 total (1 active)
Queueing time: mean = 56.913 us, max = 7.604 ms, min = 3.143 us, total = 148.430 ms
Execution time:  mean = 265.080 us, total = 691.328 ms
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1176 total (0 active), Execution time: mean = 28.731 us, total = 33.788 ms, Queueing time: mean = 117.164 us, max = 375.594 us, min = 5.071 us, total = 137.785 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 1176 total (0 active), Execution time: mean = 553.657 us, total = 651.100 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 25.972 us, total = 6.285 ms, Queueing time: mean = 40.153 us, max = 7.604 ms, min = 3.143 us, total = 9.717 ms
	GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 11.023 us, total = 143.298 us, Queueing time: mean = 64.094 us, max = 86.697 us, min = 47.656 us, total = 833.225 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 394 total (25 active)
Queueing time: mean = 56.527 us, max = 8.828 ms, min = 2.456 us, total = 22.272 ms
Execution time:  mean = 134.356 ms, total = 52.936 s
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 25.141 us, total = 6.084 ms, Queueing time: mean = 45.562 us, max = 8.828 ms, min = 2.456 us, total = 11.026 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 37 total (0 active), Execution time: mean = 30.784 us, total = 1.139 ms, Queueing time: mean = 138.328 us, max = 347.872 us, min = 5.878 us, total = 5.118 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 37 total (24 active), Execution time: mean = 1.430 s, total = 52.903 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 13 total (0 active), Execution time: mean = 109.377 us, total = 1.422 ms, Queueing time: mean = 130.402 us, max = 191.279 us, min = 44.178 us, total = 1.695 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 13 total (0 active), Execution time: mean = 657.092 us, total = 8.542 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 250 total (0 active)
Queueing time: mean = 15.984 us, max = 779.592 us, min = 621.000 ns, total = 3.996 ms
Execution time:  mean = 34.306 us, total = 8.577 ms
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 33.609 us, total = 8.133 ms, Queueing time: mean = 14.269 us, max = 779.592 us, min = 4.534 us, total = 3.453 ms
	RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 32.858 us, total = 98.574 us, Queueing time: mean = 960.333 ns, max = 1.284 us, min = 621.000 ns, total = 2.881 us
	 - 3 total (0 active), Execution time: mean = 114.255 us, total = 342.764 us, Queueing time: mean = 116.620 us, max = 131.850 us, min = 102.511 us, total = 349.861 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:49:33,583 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 24

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 25
- cumulative published bytes: 22500
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 5
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 1832 total (4 active)
Queueing time: mean = 1.288 ms, max = 251.846 ms, min = 853.000 ns, total = 2.359 s
Execution time:  mean = 408.039 us, total = 747.527 ms
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 21.475 us, total = 10.351 ms, Queueing time: mean = 39.622 us, max = 14.800 ms, min = 1.906 us, total = 19.098 ms
	GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 1.065 ms, total = 254.564 ms, Queueing time: mean = 7.389 ms, max = 251.330 ms, min = 2.247 us, total = 1.766 s
	RayletLoadPulled - 120 total (1 active), Execution time: mean = 231.999 us, total = 27.840 ms, Queueing time: mean = 71.326 us, max = 194.192 us, min = 14.896 us, total = 8.559 ms
	NodeManagerService.grpc_client.GetResourceLoad - 116 total (0 active), Execution time: mean = 1.783 ms, total = 206.826 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 116 total (0 active), Execution time: mean = 72.645 us, total = 8.427 ms, Queueing time: mean = 118.771 us, max = 2.124 ms, min = 13.688 us, total = 13.777 ms
	GcsInMemoryStore.Get - 84 total (0 active), Execution time: mean = 23.472 us, total = 1.972 ms, Queueing time: mean = 19.463 us, max = 337.749 us, min = 3.595 us, total = 1.635 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 83 total (0 active), Execution time: mean = 47.107 us, total = 3.910 ms, Queueing time: mean = 103.671 us, max = 240.717 us, min = 13.972 us, total = 8.605 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 83 total (0 active), Execution time: mean = 658.563 us, total = 54.661 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 38.782 us, total = 2.443 ms, Queueing time: mean = 108.795 us, max = 254.636 us, min = 14.068 us, total = 6.854 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 573.490 us, total = 36.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 32.126 us, total = 1.542 ms, Queueing time: mean = 105.330 us, max = 186.281 us, min = 22.398 us, total = 5.056 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 558.891 us, total = 26.827 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 41 total (1 active), Execution time: mean = 10.981 us, total = 450.240 us, Queueing time: mean = 69.871 us, max = 151.718 us, min = 19.030 us, total = 2.865 ms
	HealthCheck - 38 total (0 active), Execution time: mean = 7.884 us, total = 299.598 us, Queueing time: mean = 95.886 us, max = 174.442 us, min = 19.444 us, total = 3.644 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 665.785 us, total = 15.979 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 88.422 us, total = 2.122 ms, Queueing time: mean = 114.312 us, max = 190.945 us, min = 22.178 us, total = 2.743 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.406 ms, total = 16.875 ms, Queueing time: mean = 70.747 us, max = 137.944 us, min = 11.375 us, total = 848.965 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 104.086 us, total = 312.259 us, Queueing time: mean = 122.171 us, max = 138.303 us, min = 101.321 us, total = 366.512 us
	GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 496.070 us, total = 992.139 us, Queueing time: mean = 7.912 us, max = 15.823 us, min = 15.823 us, total = 15.823 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5380 total (1 active)
Queueing time: mean = 52.259 us, max = 7.604 ms, min = 3.138 us, total = 281.152 ms
Execution time:  mean = 249.239 us, total = 1.341 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2436 total (0 active), Execution time: mean = 25.753 us, total = 62.735 ms, Queueing time: mean = 109.832 us, max = 401.821 us, min = 5.071 us, total = 267.551 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 2436 total (0 active), Execution time: mean = 519.562 us, total = 1.266 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.363 us, total = 12.225 ms, Queueing time: mean = 24.359 us, max = 7.604 ms, min = 3.138 us, total = 11.741 ms
	GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 11.105 us, total = 277.632 us, Queueing time: mean = 70.569 us, max = 134.123 us, min = 29.930 us, total = 1.764 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 682 total (25 active)
Queueing time: mean = 39.918 us, max = 8.828 ms, min = 2.456 us, total = 27.224 ms
Execution time:  mean = 165.780 ms, total = 113.062 s
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.439 us, total = 12.262 ms, Queueing time: mean = 27.648 us, max = 8.828 ms, min = 2.456 us, total = 13.326 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 49 total (0 active), Execution time: mean = 34.354 us, total = 1.683 ms, Queueing time: mean = 132.751 us, max = 347.872 us, min = 5.878 us, total = 6.505 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 49 total (24 active), Execution time: mean = 2.306 s, total = 113.013 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 25 total (0 active), Execution time: mean = 108.082 us, total = 2.702 ms, Queueing time: mean = 118.414 us, max = 191.279 us, min = 39.531 us, total = 2.960 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 25 total (0 active), Execution time: mean = 627.921 us, total = 15.698 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:49:33,585 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 490 total (0 active)
Queueing time: mean = 13.052 us, max = 779.592 us, min = 621.000 ns, total = 6.396 ms
Execution time:  mean = 30.316 us, total = 14.855 ms
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 29.900 us, total = 14.412 ms, Queueing time: mean = 12.143 us, max = 779.592 us, min = 3.424 us, total = 5.853 ms
	RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 32.858 us, total = 98.574 us, Queueing time: mean = 960.333 ns, max = 1.284 us, min = 621.000 ns, total = 2.881 us
	 - 3 total (0 active), Execution time: mean = 114.255 us, total = 342.764 us, Queueing time: mean = 116.620 us, max = 131.850 us, min = 102.511 us, total = 349.861 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:50:33,585 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 36

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 38
- cumulative published bytes: 32718
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 8
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 3
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 2
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 2552 total (4 active)
Queueing time: mean = 935.142 us, max = 251.846 ms, min = 853.000 ns, total = 2.386 s
Execution time:  mean = 361.272 us, total = 921.965 ms
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 21.590 us, total = 15.588 ms, Queueing time: mean = 29.264 us, max = 14.800 ms, min = 1.906 us, total = 21.129 ms
	GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 761.472 us, total = 255.093 ms, Queueing time: mean = 5.283 ms, max = 251.330 ms, min = 2.247 us, total = 1.770 s
	RayletLoadPulled - 180 total (1 active), Execution time: mean = 241.374 us, total = 43.447 ms, Queueing time: mean = 70.843 us, max = 202.188 us, min = 14.896 us, total = 12.752 ms
	NodeManagerService.grpc_client.GetResourceLoad - 176 total (0 active), Execution time: mean = 1.706 ms, total = 300.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 176 total (0 active), Execution time: mean = 73.586 us, total = 12.951 ms, Queueing time: mean = 115.323 us, max = 2.124 ms, min = 13.688 us, total = 20.297 ms
	GcsInMemoryStore.Get - 96 total (0 active), Execution time: mean = 23.124 us, total = 2.220 ms, Queueing time: mean = 18.502 us, max = 337.749 us, min = 3.595 us, total = 1.776 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 95 total (0 active), Execution time: mean = 47.288 us, total = 4.492 ms, Queueing time: mean = 102.524 us, max = 240.717 us, min = 13.972 us, total = 9.740 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 95 total (0 active), Execution time: mean = 648.315 us, total = 61.590 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 39.022 us, total = 3.395 ms, Queueing time: mean = 105.842 us, max = 254.636 us, min = 14.068 us, total = 9.208 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 570.058 us, total = 49.595 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 31.547 us, total = 2.271 ms, Queueing time: mean = 103.078 us, max = 190.114 us, min = 22.398 us, total = 7.422 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 555.444 us, total = 39.992 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 61 total (1 active), Execution time: mean = 10.615 us, total = 647.532 us, Queueing time: mean = 66.860 us, max = 151.718 us, min = 19.030 us, total = 4.078 ms
	HealthCheck - 58 total (0 active), Execution time: mean = 7.832 us, total = 454.259 us, Queueing time: mean = 91.104 us, max = 199.261 us, min = 18.886 us, total = 5.284 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 666.409 us, total = 23.991 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 84.915 us, total = 3.057 ms, Queueing time: mean = 114.260 us, max = 190.945 us, min = 22.178 us, total = 4.113 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.368 ms, total = 24.631 ms, Queueing time: mean = 63.672 us, max = 137.944 us, min = 11.375 us, total = 1.146 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 124.977 us, total = 499.909 us, Queueing time: mean = 120.538 us, max = 138.303 us, min = 101.321 us, total = 482.154 us
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 926.751 us, total = 2.780 ms, Queueing time: mean = 35.602 us, max = 90.983 us, min = 15.823 us, total = 106.806 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 8152 total (1 active)
Queueing time: mean = 49.841 us, max = 7.604 ms, min = 2.691 us, total = 406.308 ms
Execution time:  mean = 244.772 us, total = 1.995 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3696 total (0 active), Execution time: mean = 24.991 us, total = 92.366 ms, Queueing time: mean = 105.370 us, max = 401.821 us, min = 5.071 us, total = 389.449 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 3696 total (0 active), Execution time: mean = 509.742 us, total = 1.884 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.751 us, total = 18.593 ms, Queueing time: mean = 19.325 us, max = 7.604 ms, min = 2.691 us, total = 13.953 ms
	GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 10.975 us, total = 406.093 us, Queueing time: mean = 75.952 us, max = 143.193 us, min = 29.930 us, total = 2.810 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 974 total (25 active)
Queueing time: mean = 32.468 us, max = 8.828 ms, min = 2.456 us, total = 31.624 ms
Execution time:  mean = 177.927 ms, total = 173.301 s
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.352 us, total = 18.304 ms, Queueing time: mean = 21.526 us, max = 8.828 ms, min = 2.456 us, total = 15.542 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 62 total (0 active), Execution time: mean = 35.252 us, total = 2.186 ms, Queueing time: mean = 121.970 us, max = 347.872 us, min = 5.878 us, total = 7.562 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 62 total (24 active), Execution time: mean = 2.794 s, total = 173.237 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 38 total (0 active), Execution time: mean = 101.689 us, total = 3.864 ms, Queueing time: mean = 107.569 us, max = 191.279 us, min = 23.555 us, total = 4.088 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 38 total (0 active), Execution time: mean = 610.504 us, total = 23.199 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 732 total (0 active)
Queueing time: mean = 11.809 us, max = 779.592 us, min = 456.000 ns, total = 8.644 ms
Execution time:  mean = 28.539 us, total = 20.890 ms
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.262 us, total = 20.405 ms, Queueing time: mean = 11.188 us, max = 779.592 us, min = 3.198 us, total = 8.078 ms
	RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 27.779 us, total = 111.116 us, Queueing time: mean = 834.250 ns, max = 1.284 us, min = 456.000 ns, total = 3.337 us
	 - 4 total (0 active), Execution time: mean = 93.099 us, total = 372.394 us, Queueing time: mean = 93.332 us, max = 131.850 us, min = 23.468 us, total = 373.329 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:51:33,586 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 48

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 52
- cumulative published bytes: 42999
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 12
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 4
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 3
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 3273 total (4 active)
Queueing time: mean = 733.655 us, max = 251.846 ms, min = 853.000 ns, total = 2.401 s
Execution time:  mean = 320.678 us, total = 1.050 s
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 20.938 us, total = 20.142 ms, Queueing time: mean = 23.807 us, max = 14.800 ms, min = 1.906 us, total = 22.902 ms
	GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 592.725 us, total = 255.465 ms, Queueing time: mean = 4.116 ms, max = 251.330 ms, min = 2.247 us, total = 1.774 s
	RayletLoadPulled - 240 total (1 active), Execution time: mean = 236.425 us, total = 56.742 ms, Queueing time: mean = 62.763 us, max = 202.188 us, min = 14.896 us, total = 15.063 ms
	NodeManagerService.grpc_client.GetResourceLoad - 236 total (0 active), Execution time: mean = 1.555 ms, total = 367.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 236 total (0 active), Execution time: mean = 70.426 us, total = 16.620 ms, Queueing time: mean = 94.973 us, max = 2.124 ms, min = 13.688 us, total = 22.414 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 516.665 us, total = 57.350 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 35.317 us, total = 3.920 ms, Queueing time: mean = 88.424 us, max = 254.636 us, min = 12.388 us, total = 9.815 ms
	GcsInMemoryStore.Get - 108 total (0 active), Execution time: mean = 22.287 us, total = 2.407 ms, Queueing time: mean = 17.361 us, max = 337.749 us, min = 3.595 us, total = 1.875 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 107 total (0 active), Execution time: mean = 46.534 us, total = 4.979 ms, Queueing time: mean = 93.984 us, max = 240.717 us, min = 13.857 us, total = 10.056 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 107 total (0 active), Execution time: mean = 612.344 us, total = 65.521 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 29.419 us, total = 2.824 ms, Queueing time: mean = 86.797 us, max = 190.114 us, min = 22.398 us, total = 8.332 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 512.000 us, total = 49.152 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 81 total (1 active), Execution time: mean = 10.000 us, total = 810.034 us, Queueing time: mean = 59.949 us, max = 151.718 us, min = 19.030 us, total = 4.856 ms
	HealthCheck - 78 total (0 active), Execution time: mean = 7.244 us, total = 565.050 us, Queueing time: mean = 76.806 us, max = 199.261 us, min = 17.698 us, total = 5.991 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 605.169 us, total = 29.048 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 80.613 us, total = 3.869 ms, Queueing time: mean = 94.495 us, max = 190.945 us, min = 22.178 us, total = 4.536 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.372 ms, total = 32.920 ms, Queueing time: mean = 57.811 us, max = 137.944 us, min = 11.375 us, total = 1.387 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	GcsResourceManager::Update - 6 total (0 active), Execution time: mean = 114.415 us, total = 686.489 us, Queueing time: mean = 123.233 us, max = 158.567 us, min = 98.679 us, total = 739.400 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.115 ms, total = 4.458 ms, Queueing time: mean = 40.390 us, max = 90.983 us, min = 15.823 us, total = 161.560 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us


[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 10924 total (1 active)
Queueing time: mean = 45.067 us, max = 7.604 ms, min = 2.691 us, total = 492.314 ms
Execution time:  mean = 226.803 us, total = 2.478 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4956 total (0 active), Execution time: mean = 23.035 us, total = 114.159 ms, Queueing time: mean = 95.402 us, max = 401.821 us, min = 4.632 us, total = 472.811 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 4956 total (0 active), Execution time: mean = 471.837 us, total = 2.338 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.466 us, total = 24.498 ms, Queueing time: mean = 16.479 us, max = 7.604 ms, min = 2.691 us, total = 15.853 ms
	GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 10.391 us, total = 509.178 us, Queueing time: mean = 72.563 us, max = 167.540 us, min = 28.413 us, total = 3.556 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1270 total (25 active)
Queueing time: mean = 27.110 us, max = 8.828 ms, min = 2.456 us, total = 34.429 ms
Execution time:  mean = 183.795 ms, total = 233.419 s
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.461 us, total = 23.532 ms, Queueing time: mean = 18.133 us, max = 8.828 ms, min = 2.456 us, total = 17.444 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 76 total (0 active), Execution time: mean = 34.893 us, total = 2.652 ms, Queueing time: mean = 104.100 us, max = 347.872 us, min = 5.878 us, total = 7.912 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 76 total (24 active), Execution time: mean = 3.070 s, total = 233.342 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 52 total (0 active), Execution time: mean = 97.773 us, total = 5.084 ms, Queueing time: mean = 89.251 us, max = 191.279 us, min = 23.555 us, total = 4.641 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 52 total (0 active), Execution time: mean = 572.702 us, total = 29.780 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 976 total (0 active)
Queueing time: mean = 10.588 us, max = 779.592 us, min = 456.000 ns, total = 10.334 ms
Execution time:  mean = 26.244 us, total = 25.614 ms
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.824 us, total = 24.843 ms, Queueing time: mean = 10.088 us, max = 779.592 us, min = 1.752 us, total = 9.704 ms
	RaySyncer.BroadcastMessage - 6 total (0 active), Execution time: mean = 24.390 us, total = 146.342 us, Queueing time: mean = 934.333 ns, max = 1.284 us, min = 456.000 ns, total = 5.606 us
	 - 6 total (0 active), Execution time: mean = 103.789 us, total = 622.734 us, Queueing time: mean = 72.303 us, max = 131.850 us, min = 23.468 us, total = 433.820 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:52:33,588 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 60

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 64
- cumulative published bytes: 52815
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 12
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 4
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 3
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:52:33,589 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 3992 total (4 active)
Queueing time: mean = 608.107 us, max = 251.846 ms, min = 853.000 ns, total = 2.428 s
Execution time:  mean = 305.390 us, total = 1.219 s
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 21.062 us, total = 25.317 ms, Queueing time: mean = 20.749 us, max = 14.800 ms, min = 1.906 us, total = 24.940 ms
	GcsInMemoryStore.Put - 527 total (0 active), Execution time: mean = 485.781 us, total = 256.006 ms, Queueing time: mean = 3.373 ms, max = 251.330 ms, min = 2.247 us, total = 1.778 s
	RayletLoadPulled - 300 total (1 active), Execution time: mean = 239.126 us, total = 71.738 ms, Queueing time: mean = 63.560 us, max = 202.188 us, min = 14.896 us, total = 19.068 ms
	NodeManagerService.grpc_client.GetResourceLoad - 296 total (0 active), Execution time: mean = 1.547 ms, total = 458.022 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 296 total (0 active), Execution time: mean = 71.344 us, total = 21.118 ms, Queueing time: mean = 96.022 us, max = 2.124 ms, min = 13.688 us, total = 28.423 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 135 total (0 active), Execution time: mean = 516.185 us, total = 69.685 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 135 total (0 active), Execution time: mean = 35.597 us, total = 4.806 ms, Queueing time: mean = 87.807 us, max = 254.636 us, min = 12.388 us, total = 11.854 ms
	GcsInMemoryStore.Get - 120 total (0 active), Execution time: mean = 22.255 us, total = 2.671 ms, Queueing time: mean = 16.840 us, max = 337.749 us, min = 3.595 us, total = 2.021 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 29.901 us, total = 3.588 ms, Queueing time: mean = 90.754 us, max = 211.127 us, min = 22.398 us, total = 10.890 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 525.557 us, total = 63.067 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 119 total (0 active), Execution time: mean = 46.698 us, total = 5.557 ms, Queueing time: mean = 92.308 us, max = 240.717 us, min = 12.541 us, total = 10.985 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 119 total (0 active), Execution time: mean = 601.147 us, total = 71.537 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 101 total (1 active), Execution time: mean = 9.856 us, total = 995.462 us, Queueing time: mean = 61.982 us, max = 151.718 us, min = 19.030 us, total = 6.260 ms
	HealthCheck - 98 total (0 active), Execution time: mean = 7.413 us, total = 726.426 us, Queueing time: mean = 81.281 us, max = 199.261 us, min = 17.698 us, total = 7.965 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 599.472 us, total = 35.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 80.139 us, total = 4.808 ms, Queueing time: mean = 92.733 us, max = 190.945 us, min = 22.178 us, total = 5.564 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.388 ms, total = 41.639 ms, Queueing time: mean = 59.311 us, max = 137.944 us, min = 11.375 us, total = 1.779 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsResourceManager::Update - 6 total (0 active), Execution time: mean = 114.415 us, total = 686.489 us, Queueing time: mean = 123.233 us, max = 158.567 us, min = 98.679 us, total = 739.400 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.235 ms, total = 6.177 ms, Queueing time: mean = 37.313 us, max = 90.983 us, min = 15.823 us, total = 186.565 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:52:33,589 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 13696 total (1 active)
Queueing time: mean = 45.412 us, max = 7.604 ms, min = 2.691 us, total = 621.963 ms
Execution time:  mean = 230.707 us, total = 3.160 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6216 total (0 active), Execution time: mean = 23.266 us, total = 144.621 ms, Queueing time: mean = 96.456 us, max = 401.821 us, min = 4.632 us, total = 599.573 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 6216 total (0 active), Execution time: mean = 480.080 us, total = 2.984 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 25.227 us, total = 30.323 ms, Queueing time: mean = 14.921 us, max = 7.604 ms, min = 2.691 us, total = 17.934 ms
	GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 10.197 us, total = 622.010 us, Queueing time: mean = 71.485 us, max = 167.540 us, min = 28.413 us, total = 4.361 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:52:33,590 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1559 total (25 active)
Queueing time: mean = 25.093 us, max = 8.828 ms, min = 2.456 us, total = 39.120 ms
Execution time:  mean = 188.304 ms, total = 293.565 s
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 24.268 us, total = 29.170 ms, Queueing time: mean = 16.278 us, max = 8.828 ms, min = 2.456 us, total = 19.566 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 88 total (0 active), Execution time: mean = 37.929 us, total = 3.338 ms, Queueing time: mean = 103.251 us, max = 347.872 us, min = 5.878 us, total = 9.086 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 88 total (24 active), Execution time: mean = 3.335 s, total = 293.473 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 64 total (0 active), Execution time: mean = 97.235 us, total = 6.223 ms, Queueing time: mean = 93.084 us, max = 191.279 us, min = 23.555 us, total = 5.957 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 64 total (0 active), Execution time: mean = 580.228 us, total = 37.135 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:52:33,590 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1216 total (0 active)
Queueing time: mean = 10.528 us, max = 779.592 us, min = 456.000 ns, total = 12.802 ms
Execution time:  mean = 26.346 us, total = 32.037 ms
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 26.012 us, total = 31.266 ms, Queueing time: mean = 10.127 us, max = 779.592 us, min = 1.752 us, total = 12.173 ms
	RaySyncer.BroadcastMessage - 6 total (0 active), Execution time: mean = 24.390 us, total = 146.342 us, Queueing time: mean = 934.333 ns, max = 1.284 us, min = 456.000 ns, total = 5.606 us
	 - 6 total (0 active), Execution time: mean = 103.789 us, total = 622.734 us, Queueing time: mean = 72.303 us, max = 131.850 us, min = 23.468 us, total = 433.820 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:53:33,590 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 72

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 77
- cumulative published bytes: 63094
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 17
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 5
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 4
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 4713 total (4 active)
Queueing time: mean = 521.136 us, max = 251.846 ms, min = 853.000 ns, total = 2.456 s
Execution time:  mean = 296.451 us, total = 1.397 s
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 21.174 us, total = 30.533 ms, Queueing time: mean = 18.727 us, max = 14.800 ms, min = 1.906 us, total = 27.004 ms
	GcsInMemoryStore.Put - 623 total (0 active), Execution time: mean = 411.851 us, total = 256.583 ms, Queueing time: mean = 2.859 ms, max = 251.330 ms, min = 2.247 us, total = 1.781 s
	RayletLoadPulled - 360 total (1 active), Execution time: mean = 243.595 us, total = 87.694 ms, Queueing time: mean = 64.708 us, max = 202.188 us, min = 14.896 us, total = 23.295 ms
	NodeManagerService.grpc_client.GetResourceLoad - 356 total (0 active), Execution time: mean = 1.555 ms, total = 553.550 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 356 total (0 active), Execution time: mean = 72.585 us, total = 25.840 ms, Queueing time: mean = 98.002 us, max = 2.124 ms, min = 13.688 us, total = 34.889 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 159 total (0 active), Execution time: mean = 522.259 us, total = 83.039 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 159 total (0 active), Execution time: mean = 35.946 us, total = 5.715 ms, Queueing time: mean = 90.140 us, max = 254.636 us, min = 12.388 us, total = 14.332 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 30.488 us, total = 4.390 ms, Queueing time: mean = 94.624 us, max = 216.808 us, min = 22.398 us, total = 13.626 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 529.306 us, total = 76.220 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 132 total (0 active), Execution time: mean = 22.371 us, total = 2.953 ms, Queueing time: mean = 16.390 us, max = 337.749 us, min = 3.595 us, total = 2.164 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 131 total (0 active), Execution time: mean = 46.457 us, total = 6.086 ms, Queueing time: mean = 93.693 us, max = 240.717 us, min = 12.541 us, total = 12.274 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 131 total (0 active), Execution time: mean = 599.666 us, total = 78.556 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 121 total (1 active), Execution time: mean = 9.932 us, total = 1.202 ms, Queueing time: mean = 63.858 us, max = 151.718 us, min = 19.030 us, total = 7.727 ms
	HealthCheck - 118 total (0 active), Execution time: mean = 7.616 us, total = 898.646 us, Queueing time: mean = 84.027 us, max = 199.261 us, min = 17.698 us, total = 9.915 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 610.827 us, total = 43.980 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 80.590 us, total = 5.802 ms, Queueing time: mean = 95.639 us, max = 190.945 us, min = 22.178 us, total = 6.886 ms
	GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.399 ms, total = 50.379 ms, Queueing time: mean = 60.391 us, max = 137.944 us, min = 11.375 us, total = 2.174 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsResourceManager::Update - 8 total (0 active), Execution time: mean = 112.004 us, total = 896.028 us, Queueing time: mean = 126.100 us, max = 158.567 us, min = 98.679 us, total = 1.009 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.308 ms, total = 7.849 ms, Queueing time: mean = 40.277 us, max = 90.983 us, min = 15.823 us, total = 241.663 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 16468 total (1 active)
Queueing time: mean = 46.567 us, max = 7.604 ms, min = 2.691 us, total = 766.869 ms
Execution time:  mean = 236.090 us, total = 3.888 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7476 total (0 active), Execution time: mean = 23.763 us, total = 177.651 ms, Queueing time: mean = 99.132 us, max = 401.821 us, min = 4.632 us, total = 741.110 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 7476 total (0 active), Execution time: mean = 491.289 us, total = 3.673 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 25.411 us, total = 36.643 ms, Queueing time: mean = 14.011 us, max = 7.604 ms, min = 2.691 us, total = 20.204 ms
	GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 10.272 us, total = 749.880 us, Queueing time: mean = 74.801 us, max = 167.540 us, min = 28.413 us, total = 5.460 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1851 total (25 active)
Queueing time: mean = 23.903 us, max = 8.828 ms, min = 2.456 us, total = 44.244 ms
Execution time:  mean = 191.103 ms, total = 353.732 s
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 24.248 us, total = 34.965 ms, Queueing time: mean = 15.082 us, max = 8.828 ms, min = 2.456 us, total = 21.748 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 101 total (0 active), Execution time: mean = 39.215 us, total = 3.961 ms, Queueing time: mean = 103.415 us, max = 347.872 us, min = 5.878 us, total = 10.445 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 101 total (24 active), Execution time: mean = 3.501 s, total = 353.622 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 77 total (0 active), Execution time: mean = 101.615 us, total = 7.824 ms, Queueing time: mean = 97.939 us, max = 191.279 us, min = 23.555 us, total = 7.541 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 77 total (0 active), Execution time: mean = 602.516 us, total = 46.394 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1460 total (0 active)
Queueing time: mean = 10.580 us, max = 779.592 us, min = 456.000 ns, total = 15.447 ms
Execution time:  mean = 26.493 us, total = 38.680 ms
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 26.048 us, total = 37.562 ms, Queueing time: mean = 10.090 us, max = 779.592 us, min = 1.752 us, total = 14.550 ms
	RaySyncer.BroadcastMessage - 8 total (0 active), Execution time: mean = 26.053 us, total = 208.422 us, Queueing time: mean = 965.875 ns, max = 1.284 us, min = 456.000 ns, total = 7.727 us
	 - 8 total (0 active), Execution time: mean = 113.459 us, total = 907.670 us, Queueing time: mean = 87.368 us, max = 146.934 us, min = 23.468 us, total = 698.947 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:54:33,592 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 84

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 90
- cumulative published bytes: 72943
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 17
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 5
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 4
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 5432 total (4 active)
Queueing time: mean = 457.183 us, max = 251.846 ms, min = 853.000 ns, total = 2.483 s
Execution time:  mean = 289.700 us, total = 1.574 s
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 21.243 us, total = 35.731 ms, Queueing time: mean = 17.260 us, max = 14.800 ms, min = 1.906 us, total = 29.032 ms
	GcsInMemoryStore.Put - 719 total (0 active), Execution time: mean = 357.672 us, total = 257.166 ms, Queueing time: mean = 2.483 ms, max = 251.330 ms, min = 2.247 us, total = 1.785 s
	RayletLoadPulled - 420 total (1 active), Execution time: mean = 246.664 us, total = 103.599 ms, Queueing time: mean = 65.672 us, max = 202.188 us, min = 14.896 us, total = 27.582 ms
	NodeManagerService.grpc_client.GetResourceLoad - 416 total (0 active), Execution time: mean = 1.561 ms, total = 649.361 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 416 total (0 active), Execution time: mean = 73.159 us, total = 30.434 ms, Queueing time: mean = 98.309 us, max = 2.124 ms, min = 13.688 us, total = 40.897 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 183 total (0 active), Execution time: mean = 524.436 us, total = 95.972 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 183 total (0 active), Execution time: mean = 36.461 us, total = 6.672 ms, Queueing time: mean = 90.924 us, max = 254.636 us, min = 12.388 us, total = 16.639 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 168 total (0 active), Execution time: mean = 30.483 us, total = 5.121 ms, Queueing time: mean = 95.312 us, max = 216.808 us, min = 22.398 us, total = 16.012 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 168 total (0 active), Execution time: mean = 524.842 us, total = 88.174 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 144 total (0 active), Execution time: mean = 22.805 us, total = 3.284 ms, Queueing time: mean = 16.054 us, max = 337.749 us, min = 3.595 us, total = 2.312 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 143 total (0 active), Execution time: mean = 47.273 us, total = 6.760 ms, Queueing time: mean = 93.956 us, max = 240.717 us, min = 12.541 us, total = 13.436 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 143 total (0 active), Execution time: mean = 598.521 us, total = 85.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 141 total (1 active), Execution time: mean = 9.873 us, total = 1.392 ms, Queueing time: mean = 64.648 us, max = 151.718 us, min = 16.805 us, total = 9.115 ms
	HealthCheck - 138 total (0 active), Execution time: mean = 7.684 us, total = 1.060 ms, Queueing time: mean = 87.606 us, max = 199.261 us, min = 17.698 us, total = 12.090 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 84 total (0 active), Execution time: mean = 615.923 us, total = 51.738 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 84 total (0 active), Execution time: mean = 82.166 us, total = 6.902 ms, Queueing time: mean = 97.472 us, max = 190.945 us, min = 22.178 us, total = 8.188 ms
	GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), Execution time: mean = 1.413 ms, total = 59.347 ms, Queueing time: mean = 62.353 us, max = 137.944 us, min = 11.375 us, total = 2.619 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsResourceManager::Update - 8 total (0 active), Execution time: mean = 112.004 us, total = 896.028 us, Queueing time: mean = 126.100 us, max = 158.567 us, min = 98.679 us, total = 1.009 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), Execution time: mean = 1.350 ms, total = 9.448 ms, Queueing time: mean = 41.430 us, max = 90.983 us, min = 15.823 us, total = 290.007 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 19240 total (1 active)
Queueing time: mean = 47.331 us, max = 7.604 ms, min = 2.691 us, total = 910.652 ms
Execution time:  mean = 239.598 us, total = 4.610 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 8736 total (0 active), Execution time: mean = 24.309 us, total = 212.364 ms, Queueing time: mean = 100.961 us, max = 401.821 us, min = 4.632 us, total = 882.000 ms
	TaskInfoGcsService.grpc_server.AddTaskEventData - 8736 total (0 active), Execution time: mean = 498.401 us, total = 4.354 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 25.298 us, total = 42.551 ms, Queueing time: mean = 13.228 us, max = 7.604 ms, min = 2.691 us, total = 22.250 ms
	GcsTaskManager.GcJobSummary - 85 total (1 active), Execution time: mean = 10.596 us, total = 900.685 us, Queueing time: mean = 74.208 us, max = 167.540 us, min = 28.413 us, total = 6.308 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 2143 total (25 active)
Queueing time: mean = 23.000 us, max = 8.828 ms, min = 2.456 us, total = 49.289 ms
Execution time:  mean = 193.141 ms, total = 413.902 s
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 24.331 us, total = 40.925 ms, Queueing time: mean = 14.244 us, max = 8.828 ms, min = 2.456 us, total = 23.959 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 114 total (0 active), Execution time: mean = 39.182 us, total = 4.467 ms, Queueing time: mean = 105.139 us, max = 347.872 us, min = 5.878 us, total = 11.986 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 114 total (24 active), Execution time: mean = 3.630 s, total = 413.777 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 90 total (0 active), Execution time: mean = 100.418 us, total = 9.038 ms, Queueing time: mean = 98.157 us, max = 191.279 us, min = 23.555 us, total = 8.834 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 90 total (0 active), Execution time: mean = 603.694 us, total = 54.332 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1700 total (0 active)
Queueing time: mean = 10.510 us, max = 779.592 us, min = 456.000 ns, total = 17.867 ms
Execution time:  mean = 26.541 us, total = 45.120 ms
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 26.161 us, total = 44.002 ms, Queueing time: mean = 10.089 us, max = 779.592 us, min = 1.752 us, total = 16.970 ms
	RaySyncer.BroadcastMessage - 8 total (0 active), Execution time: mean = 26.053 us, total = 208.422 us, Queueing time: mean = 965.875 ns, max = 1.284 us, min = 456.000 ns, total = 7.727 us
	 - 8 total (0 active), Execution time: mean = 113.459 us, total = 907.670 us, Queueing time: mean = 87.368 us, max = 146.934 us, min = 23.468 us, total = 698.947 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:55:33,593 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 96

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 103
- cumulative published bytes: 83131
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 6152 total (4 active)
Queueing time: mean = 408.569 us, max = 251.846 ms, min = 853.000 ns, total = 2.514 s
Execution time:  mean = 284.953 us, total = 1.753 s
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 21.302 us, total = 40.942 ms, Queueing time: mean = 16.172 us, max = 14.800 ms, min = 1.906 us, total = 31.082 ms
	GcsInMemoryStore.Put - 815 total (0 active), Execution time: mean = 316.264 us, total = 257.755 ms, Queueing time: mean = 2.196 ms, max = 251.330 ms, min = 2.247 us, total = 1.790 s
	RayletLoadPulled - 480 total (1 active), Execution time: mean = 249.454 us, total = 119.738 ms, Queueing time: mean = 66.387 us, max = 202.188 us, min = 14.896 us, total = 31.866 ms
	NodeManagerService.grpc_client.GetResourceLoad - 476 total (0 active), Execution time: mean = 1.565 ms, total = 744.792 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 476 total (0 active), Execution time: mean = 73.566 us, total = 35.018 ms, Queueing time: mean = 100.260 us, max = 2.124 ms, min = 13.688 us, total = 47.724 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 207 total (0 active), Execution time: mean = 531.626 us, total = 110.047 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 207 total (0 active), Execution time: mean = 37.358 us, total = 7.733 ms, Queueing time: mean = 92.613 us, max = 254.636 us, min = 12.388 us, total = 19.171 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 192 total (0 active), Execution time: mean = 30.339 us, total = 5.825 ms, Queueing time: mean = 97.276 us, max = 216.808 us, min = 22.398 us, total = 18.677 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 192 total (0 active), Execution time: mean = 526.057 us, total = 101.003 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 161 total (1 active), Execution time: mean = 9.795 us, total = 1.577 ms, Queueing time: mean = 65.945 us, max = 151.718 us, min = 16.805 us, total = 10.617 ms
	HealthCheck - 158 total (0 active), Execution time: mean = 7.734 us, total = 1.222 ms, Queueing time: mean = 89.494 us, max = 199.261 us, min = 17.698 us, total = 14.140 ms
	GcsInMemoryStore.Get - 156 total (0 active), Execution time: mean = 23.030 us, total = 3.593 ms, Queueing time: mean = 15.772 us, max = 337.749 us, min = 3.595 us, total = 2.460 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 155 total (0 active), Execution time: mean = 47.539 us, total = 7.368 ms, Queueing time: mean = 95.770 us, max = 240.717 us, min = 12.541 us, total = 14.844 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 155 total (0 active), Execution time: mean = 599.794 us, total = 92.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 96 total (0 active), Execution time: mean = 83.684 us, total = 8.034 ms, Queueing time: mean = 100.710 us, max = 190.945 us, min = 22.178 us, total = 9.668 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 96 total (0 active), Execution time: mean = 626.306 us, total = 60.125 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), Execution time: mean = 1.423 ms, total = 68.282 ms, Queueing time: mean = 63.136 us, max = 137.944 us, min = 11.375 us, total = 3.031 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), Execution time: mean = 1.374 ms, total = 10.996 ms, Queueing time: mean = 45.210 us, max = 90.983 us, min = 15.823 us, total = 361.682 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 22012 total (1 active)
Queueing time: mean = 47.734 us, max = 7.604 ms, min = 2.691 us, total = 1.051 s
Execution time:  mean = 242.040 us, total = 5.328 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 9996 total (0 active), Execution time: mean = 24.583 us, total = 245.727 ms, Queueing time: mean = 101.952 us, max = 401.821 us, min = 4.632 us, total = 1.019 s
	TaskInfoGcsService.grpc_server.AddTaskEventData - 9996 total (0 active), Execution time: mean = 503.448 us, total = 5.032 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 25.266 us, total = 48.562 ms, Queueing time: mean = 12.642 us, max = 7.604 ms, min = 2.691 us, total = 24.298 ms
	GcsTaskManager.GcJobSummary - 97 total (1 active), Execution time: mean = 10.483 us, total = 1.017 ms, Queueing time: mean = 74.521 us, max = 167.540 us, min = 28.413 us, total = 7.229 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 2435 total (25 active)
Queueing time: mean = 22.300 us, max = 8.828 ms, min = 2.456 us, total = 54.300 ms
Execution time:  mean = 194.687 ms, total = 474.062 s
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 24.684 us, total = 47.443 ms, Queueing time: mean = 13.658 us, max = 8.828 ms, min = 2.456 us, total = 26.251 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 127 total (0 active), Execution time: mean = 40.539 us, total = 5.148 ms, Queueing time: mean = 105.201 us, max = 347.872 us, min = 5.878 us, total = 13.360 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 127 total (24 active), Execution time: mean = 3.732 s, total = 473.920 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 103 total (0 active), Execution time: mean = 102.970 us, total = 10.606 ms, Queueing time: mean = 98.814 us, max = 191.279 us, min = 23.555 us, total = 10.178 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 103 total (0 active), Execution time: mean = 609.231 us, total = 62.751 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:55:33,595 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1942 total (0 active)
Queueing time: mean = 10.490 us, max = 779.592 us, min = 456.000 ns, total = 20.372 ms
Execution time:  mean = 26.623 us, total = 51.703 ms
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 26.238 us, total = 50.429 ms, Queueing time: mean = 10.072 us, max = 779.592 us, min = 1.752 us, total = 19.359 ms
	RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
	 - 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:56:33,595 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 108

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 115
- cumulative published bytes: 92947
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 6871 total (4 active)
Queueing time: mean = 369.918 us, max = 251.846 ms, min = 853.000 ns, total = 2.542 s
Execution time:  mean = 280.558 us, total = 1.928 s
Event stats:
	event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 21.206 us, total = 45.847 ms, Queueing time: mean = 15.280 us, max = 14.800 ms, min = 1.906 us, total = 33.035 ms
	GcsInMemoryStore.Put - 911 total (0 active), Execution time: mean = 283.559 us, total = 258.323 ms, Queueing time: mean = 1.968 ms, max = 251.330 ms, min = 2.247 us, total = 1.793 s
	RayletLoadPulled - 540 total (1 active), Execution time: mean = 251.835 us, total = 135.991 ms, Queueing time: mean = 66.612 us, max = 202.188 us, min = 14.896 us, total = 35.971 ms
	NodeManagerService.grpc_client.GetResourceLoad - 536 total (0 active), Execution time: mean = 1.567 ms, total = 839.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 536 total (0 active), Execution time: mean = 74.560 us, total = 39.964 ms, Queueing time: mean = 101.427 us, max = 2.124 ms, min = 13.688 us, total = 54.365 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 231 total (0 active), Execution time: mean = 532.666 us, total = 123.046 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 231 total (0 active), Execution time: mean = 37.260 us, total = 8.607 ms, Queueing time: mean = 93.995 us, max = 254.636 us, min = 12.388 us, total = 21.713 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 216 total (0 active), Execution time: mean = 30.408 us, total = 6.568 ms, Queueing time: mean = 98.657 us, max = 216.808 us, min = 22.398 us, total = 21.310 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 216 total (0 active), Execution time: mean = 522.375 us, total = 112.833 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 181 total (1 active), Execution time: mean = 9.873 us, total = 1.787 ms, Queueing time: mean = 66.131 us, max = 151.718 us, min = 16.805 us, total = 11.970 ms
	HealthCheck - 178 total (0 active), Execution time: mean = 7.732 us, total = 1.376 ms, Queueing time: mean = 90.008 us, max = 199.261 us, min = 17.698 us, total = 16.021 ms
	GcsInMemoryStore.Get - 168 total (0 active), Execution time: mean = 23.814 us, total = 4.001 ms, Queueing time: mean = 15.568 us, max = 337.749 us, min = 3.595 us, total = 2.616 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 167 total (0 active), Execution time: mean = 47.880 us, total = 7.996 ms, Queueing time: mean = 96.127 us, max = 240.717 us, min = 12.541 us, total = 16.053 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 167 total (0 active), Execution time: mean = 598.527 us, total = 99.954 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 108 total (0 active), Execution time: mean = 83.147 us, total = 8.980 ms, Queueing time: mean = 104.636 us, max = 211.974 us, min = 22.178 us, total = 11.301 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 108 total (0 active), Execution time: mean = 629.266 us, total = 67.961 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 54 total (1 active), Execution time: mean = 1.410 ms, total = 76.161 ms, Queueing time: mean = 61.739 us, max = 137.944 us, min = 11.375 us, total = 3.334 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 9 total (1 active, 1 running), Execution time: mean = 1.388 ms, total = 12.493 ms, Queueing time: mean = 45.555 us, max = 90.983 us, min = 15.823 us, total = 409.992 us
	GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 24784 total (1 active)
Queueing time: mean = 47.600 us, max = 7.604 ms, min = 2.691 us, total = 1.180 s
Execution time:  mean = 242.233 us, total = 6.003 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 11256 total (0 active), Execution time: mean = 24.597 us, total = 276.865 ms, Queueing time: mean = 101.741 us, max = 401.821 us, min = 4.632 us, total = 1.145 s
	TaskInfoGcsService.grpc_server.AddTaskEventData - 11256 total (0 active), Execution time: mean = 503.811 us, total = 5.671 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 25.249 us, total = 54.589 ms, Queueing time: mean = 12.189 us, max = 7.604 ms, min = 2.691 us, total = 26.352 ms
	GcsTaskManager.GcJobSummary - 109 total (1 active), Execution time: mean = 10.302 us, total = 1.123 ms, Queueing time: mean = 74.094 us, max = 167.540 us, min = 28.413 us, total = 8.076 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 2723 total (25 active)
Queueing time: mean = 21.508 us, max = 8.828 ms, min = 2.456 us, total = 58.567 ms
Execution time:  mean = 196.200 ms, total = 534.251 s
Event stats:
	event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 24.706 us, total = 53.414 ms, Queueing time: mean = 13.192 us, max = 8.828 ms, min = 2.456 us, total = 28.520 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 139 total (0 active), Execution time: mean = 40.437 us, total = 5.621 ms, Queueing time: mean = 102.308 us, max = 347.872 us, min = 5.878 us, total = 14.221 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 139 total (24 active), Execution time: mean = 3.842 s, total = 534.095 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 115 total (0 active), Execution time: mean = 102.425 us, total = 11.779 ms, Queueing time: mean = 98.395 us, max = 191.279 us, min = 23.555 us, total = 11.315 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 115 total (0 active), Execution time: mean = 604.014 us, total = 69.462 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 2182 total (0 active)
Queueing time: mean = 10.447 us, max = 779.592 us, min = 456.000 ns, total = 22.795 ms
Execution time:  mean = 26.748 us, total = 58.365 ms
Event stats:
	event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 26.406 us, total = 57.090 ms, Queueing time: mean = 10.075 us, max = 779.592 us, min = 1.752 us, total = 21.782 ms
	RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
	 - 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:57:33,597 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 120

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 127
- cumulative published bytes: 102765
- current buffered bytes: 819
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 7590 total (4 active)
Queueing time: mean = 338.039 us, max = 251.846 ms, min = 853.000 ns, total = 2.566 s
Execution time:  mean = 274.599 us, total = 2.084 s
Event stats:
	event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 21.158 us, total = 50.822 ms, Queueing time: mean = 14.559 us, max = 14.800 ms, min = 1.906 us, total = 34.971 ms
	GcsInMemoryStore.Put - 1007 total (0 active), Execution time: mean = 257.012 us, total = 258.811 ms, Queueing time: mean = 1.785 ms, max = 251.330 ms, min = 2.247 us, total = 1.797 s
	RayletLoadPulled - 600 total (1 active), Execution time: mean = 252.532 us, total = 151.519 ms, Queueing time: mean = 66.206 us, max = 202.188 us, min = 14.896 us, total = 39.724 ms
	NodeManagerService.grpc_client.GetResourceLoad - 596 total (0 active), Execution time: mean = 1.547 ms, total = 921.735 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 596 total (0 active), Execution time: mean = 74.630 us, total = 44.479 ms, Queueing time: mean = 99.525 us, max = 2.124 ms, min = 13.688 us, total = 59.317 ms
	InternalKVGcsService.grpc_server.InternalKVPut - 255 total (0 active), Execution time: mean = 526.294 us, total = 134.205 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 255 total (0 active), Execution time: mean = 36.566 us, total = 9.324 ms, Queueing time: mean = 91.577 us, max = 254.636 us, min = 12.388 us, total = 23.352 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 240 total (0 active), Execution time: mean = 30.998 us, total = 7.439 ms, Queueing time: mean = 97.956 us, max = 216.808 us, min = 14.918 us, total = 23.509 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 240 total (0 active), Execution time: mean = 516.369 us, total = 123.928 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 201 total (1 active), Execution time: mean = 9.748 us, total = 1.959 ms, Queueing time: mean = 65.449 us, max = 151.718 us, min = 16.805 us, total = 13.155 ms
	HealthCheck - 198 total (0 active), Execution time: mean = 7.693 us, total = 1.523 ms, Queueing time: mean = 89.297 us, max = 199.261 us, min = 17.136 us, total = 17.681 ms
	GcsInMemoryStore.Get - 180 total (0 active), Execution time: mean = 23.661 us, total = 4.259 ms, Queueing time: mean = 15.251 us, max = 337.749 us, min = 3.595 us, total = 2.745 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 179 total (0 active), Execution time: mean = 47.566 us, total = 8.514 ms, Queueing time: mean = 96.182 us, max = 240.717 us, min = 12.541 us, total = 17.217 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 179 total (0 active), Execution time: mean = 593.606 us, total = 106.255 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 120 total (0 active), Execution time: mean = 82.496 us, total = 9.899 ms, Queueing time: mean = 103.536 us, max = 211.974 us, min = 21.535 us, total = 12.424 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 120 total (0 active), Execution time: mean = 624.939 us, total = 74.993 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 60 total (1 active), Execution time: mean = 1.403 ms, total = 84.209 ms, Queueing time: mean = 62.894 us, max = 187.024 us, min = 11.375 us, total = 3.774 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 10 total (1 active, 1 running), Execution time: mean = 1.432 ms, total = 14.315 ms, Queueing time: mean = 47.371 us, max = 90.983 us, min = 15.823 us, total = 473.711 us
	GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us


[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 27556 total (1 active)
Queueing time: mean = 46.976 us, max = 7.604 ms, min = 2.473 us, total = 1.294 s
Execution time:  mean = 239.789 us, total = 6.608 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 12516 total (0 active), Execution time: mean = 24.311 us, total = 304.279 ms, Queueing time: mean = 100.456 us, max = 401.821 us, min = 4.212 us, total = 1.257 s
	TaskInfoGcsService.grpc_server.AddTaskEventData - 12516 total (0 active), Execution time: mean = 498.698 us, total = 6.242 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 25.159 us, total = 60.433 ms, Queueing time: mean = 11.820 us, max = 7.604 ms, min = 2.473 us, total = 28.393 ms
	GcsTaskManager.GcJobSummary - 121 total (1 active), Execution time: mean = 10.059 us, total = 1.217 ms, Queueing time: mean = 71.671 us, max = 167.540 us, min = 25.713 us, total = 8.672 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us


[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 3058 total (25 active)
Queueing time: mean = 21.363 us, max = 8.828 ms, min = 2.456 us, total = 65.329 ms
Execution time:  mean = 4.674 s, total = 14294.617 s
Event stats:
	event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 24.610 us, total = 59.112 ms, Queueing time: mean = 12.748 us, max = 8.828 ms, min = 2.456 us, total = 30.620 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 174 total (0 active), Execution time: mean = 37.518 us, total = 6.528 ms, Queueing time: mean = 101.571 us, max = 347.872 us, min = 5.878 us, total = 17.673 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 174 total (24 active), Execution time: mean = 82.152 s, total = 14294.445 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 127 total (0 active), Execution time: mean = 101.531 us, total = 12.894 ms, Queueing time: mean = 98.114 us, max = 200.459 us, min = 23.555 us, total = 12.461 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 127 total (0 active), Execution time: mean = 599.919 us, total = 76.190 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
	Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 237.114 us, total = 711.342 us, Queueing time: mean = 47.608 us, max = 77.683 us, min = 65.142 us, total = 142.825 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms


[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 2422 total (0 active)
Queueing time: mean = 10.347 us, max = 779.592 us, min = 456.000 ns, total = 25.061 ms
Execution time:  mean = 26.511 us, total = 64.210 ms
Event stats:
	event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 26.201 us, total = 62.936 ms, Queueing time: mean = 10.012 us, max = 779.592 us, min = 1.752 us, total = 24.048 ms
	RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
	 - 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us


[2025-01-21 05:58:01,517 I 18551 18551] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,799 I 18551 18600] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,799 I 18551 18551] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-21 05:58:01,799 I 18551 18600] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,801 I 18551 18551] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-21 05:58:01,931 I 18551 18551] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-21 05:58:01,932 I 18551 18551] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-21 05:58:02,048 I 18551 18551] (gcs_server) stats.h:120: Stats module has shutdown.