ray/session_2025-01-20_22-53-01_396103_8700/logs/gcs_server.out

[2025-01-20 22:53:01,457 I 11638 11638] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-20 22:53:01,458 I 11638 11638] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-20 22:53:01,463 I 11638 11638] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-20 22:53:01,463 I 11638 11638] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-20 22:53:01,463 I 11638 11638] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-20 22:53:01,463 I 11638 11638] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-20 22:53:01,463 I 11638 11638] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-20 22:53:01,471 I 11638 11638] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 9ba956de09a13ad1bbed4734328a99f6ee8647c3b5e312c581e20f93
[2025-01-20 22:53:01,473 I 11638 11638] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-20 22:53:01,476 I 11638 11638] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 59305.
[2025-01-20 22:53:01,732 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:53:01,732 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 92.978 ms, max = 256.944 ms, min = 1.392 us, total = 2.324 s
Execution time:  mean = 10.372 ms, total = 259.288 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 28.551 ms, total = 256.961 ms, Queueing time: mean = 199.373 ms, max = 256.536 ms, min = 1.392 us, total = 1.794 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 6.578 us, total = 32.889 us, Queueing time: mean = 43.462 us, max = 46.574 us, min = 40.250 us, total = 217.308 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.662 us, total = 10.648 us, Queueing time: mean = 128.447 ms, max = 256.944 ms, min = 256.844 ms, total = 513.788 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 7.654 us, total = 15.307 us, Queueing time: mean = 7.504 ms, max = 14.831 ms, min = 177.649 us, total = 15.009 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 2.259 ms, total = 2.259 ms, Queueing time: mean = 1.075 ms, max = 1.075 ms, min = 1.075 ms, total = 1.075 ms
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 2.273 us, max = 2.273 us, min = 2.273 us, total = 2.273 us


[2025-01-20 22:53:01,732 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 803.740 us, max = 3.876 ms, min = 9.230 us, total = 4.019 ms
Execution time:  mean = 173.146 us, total = 865.731 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 283.910 us, total = 851.729 us, Queueing time: mean = 1.307 ms, max = 3.876 ms, min = 9.230 us, total = 3.920 ms
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:53:01,732 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 170.839 us, max = 531.471 us, min = 11.020 us, total = 854.197 us
Execution time:  mean = 729.408 us, total = 3.647 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.207 ms, total = 3.621 ms, Queueing time: mean = 247.043 us, max = 531.471 us, min = 11.020 us, total = 741.130 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:53:01,732 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 5 total (0 active)
Queueing time: mean = 2.251 ms, max = 10.161 ms, min = 9.417 us, total = 11.254 ms
Execution time:  mean = 53.616 us, total = 268.080 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 88.769 us, total = 266.308 us, Queueing time: mean = 3.406 ms, max = 10.161 ms, min = 9.417 us, total = 10.217 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 22:53:03,876 I 11638 11638] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 22:53:03,876 I 11638 11638] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 22:53:03,877 I 11638 11638] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf registered, will try to reschedule all the infeasible placement groups.
[2025-01-20 22:53:03,883 I 11638 11705] (gcs_server) ray_syncer.cc:377: Get connection node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 22:53:04,799 I 11638 11638] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 8700
[2025-01-20 22:53:04,799 I 11638 11638] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 8700
[2025-01-20 22:53:11,476 W 11638 11640] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:56214: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-20 22:54:01,733 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 12

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 13
- cumulative published bytes: 11234
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 1
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 1
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:54:01,734 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 1104 total (4 active)
Queueing time: mean = 2.185 ms, max = 256.944 ms, min = 761.000 ns, total = 2.412 s
Execution time:  mean = 511.760 us, total = 564.983 ms
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 28.446 us, total = 6.884 ms, Queueing time: mean = 73.001 us, max = 14.831 ms, min = 4.053 us, total = 17.666 ms
	GcsInMemoryStore.Put - 141 total (0 active), Execution time: mean = 1.835 ms, total = 258.715 ms, Queueing time: mean = 12.765 ms, max = 256.536 ms, min = 1.392 us, total = 1.800 s
	GcsInMemoryStore.Get - 71 total (0 active), Execution time: mean = 23.799 us, total = 1.690 ms, Queueing time: mean = 17.038 us, max = 74.560 us, min = 2.273 us, total = 1.210 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 70 total (0 active), Execution time: mean = 46.394 us, total = 3.248 ms, Queueing time: mean = 102.387 us, max = 273.037 us, min = 3.930 us, total = 7.167 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 70 total (0 active), Execution time: mean = 635.077 us, total = 44.455 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	RayletLoadPulled - 60 total (1 active), Execution time: mean = 272.259 us, total = 16.336 ms, Queueing time: mean = 364.101 us, max = 16.997 ms, min = 43.541 us, total = 21.846 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 86.461 us, total = 4.928 ms, Queueing time: mean = 109.199 us, max = 173.752 us, min = 26.249 us, total = 6.224 ms
	NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.574 ms, total = 89.718 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 37 total (0 active), Execution time: mean = 631.198 us, total = 23.354 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 37 total (0 active), Execution time: mean = 44.694 us, total = 1.654 ms, Queueing time: mean = 109.988 us, max = 151.188 us, min = 37.537 us, total = 4.070 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 31.978 us, total = 767.480 us, Queueing time: mean = 126.292 us, max = 239.706 us, min = 32.077 us, total = 3.031 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 605.744 us, total = 14.538 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 10.813 us, total = 216.256 us, Queueing time: mean = 927.529 us, max = 17.165 ms, min = 42.301 us, total = 18.551 ms
	HealthCheck - 18 total (0 active), Execution time: mean = 9.767 us, total = 175.807 us, Queueing time: mean = 107.527 us, max = 136.632 us, min = 50.053 us, total = 1.935 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 96.464 us, total = 1.158 ms, Queueing time: mean = 140.109 us, max = 183.596 us, min = 105.168 us, total = 1.681 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 756.932 us, total = 9.083 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.291 ms, total = 7.744 ms, Queueing time: mean = 66.823 us, max = 94.127 us, min = 65.083 us, total = 400.938 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	GcsResourceManager::Update - 1 total (0 active), Execution time: mean = 94.327 us, total = 94.327 us, Queueing time: mean = 174.838 us, max = 174.838 us, min = 174.838 us, total = 174.838 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:54:01,734 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 2664 total (2 active)
Queueing time: mean = 54.797 us, max = 3.876 ms, min = 3.153 us, total = 145.979 ms
Execution time:  mean = 267.311 us, total = 712.116 ms
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 1204 total (1 active), Execution time: mean = 556.910 us, total = 670.520 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1204 total (0 active), Execution time: mean = 27.996 us, total = 33.707 ms, Queueing time: mean = 115.138 us, max = 295.599 us, min = 7.657 us, total = 138.626 ms
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 32.000 us, total = 7.744 ms, Queueing time: mean = 25.984 us, max = 3.876 ms, min = 3.153 us, total = 6.288 ms
	GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 10.036 us, total = 130.468 us, Queueing time: mean = 74.389 us, max = 115.736 us, min = 61.345 us, total = 967.054 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:54:01,734 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 394 total (25 active)
Queueing time: mean = 29.978 us, max = 531.471 us, min = 4.219 us, total = 11.811 ms
Execution time:  mean = 136.224 ms, total = 53.672 s
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 39.706 us, total = 9.609 ms, Queueing time: mean = 11.982 us, max = 531.471 us, min = 4.219 us, total = 2.900 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 37 total (0 active), Execution time: mean = 31.118 us, total = 1.151 ms, Queueing time: mean = 126.642 us, max = 279.534 us, min = 27.936 us, total = 4.686 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 37 total (24 active), Execution time: mean = 1.450 s, total = 53.634 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 13 total (0 active), Execution time: mean = 106.031 us, total = 1.378 ms, Queueing time: mean = 118.116 us, max = 136.962 us, min = 76.884 us, total = 1.536 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 13 total (0 active), Execution time: mean = 667.391 us, total = 8.676 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:54:01,734 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 246 total (0 active)
Queueing time: mean = 54.839 us, max = 10.161 ms, min = 362.000 ns, total = 13.490 ms
Execution time:  mean = 24.827 us, total = 6.107 ms
Event stats:
	event_loop_lag_probe - 242 total (0 active), Execution time: mean = 24.695 us, total = 5.976 ms, Queueing time: mean = 50.951 us, max = 10.161 ms, min = 4.662 us, total = 12.330 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms
	RaySyncer.BroadcastMessage - 1 total (0 active), Execution time: mean = 25.880 us, total = 25.880 us, Queueing time: mean = 362.000 ns, max = 362.000 ns, min = 362.000 ns, total = 362.000 ns
	 - 1 total (0 active), Execution time: mean = 103.626 us, total = 103.626 us, Queueing time: mean = 123.039 us, max = 123.039 us, min = 123.039 us, total = 123.039 us


[2025-01-20 22:55:01,734 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 24

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 35
- cumulative published bytes: 23501
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 30
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:55:01,735 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 1863 total (4 active)
Queueing time: mean = 1.313 ms, max = 256.944 ms, min = 761.000 ns, total = 2.446 s
Execution time:  mean = 402.953 us, total = 750.701 ms
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 26.437 us, total = 12.743 ms, Queueing time: mean = 41.469 us, max = 14.831 ms, min = 4.053 us, total = 19.988 ms
	GcsInMemoryStore.Put - 238 total (0 active), Execution time: mean = 1.090 ms, total = 259.382 ms, Queueing time: mean = 7.580 ms, max = 256.536 ms, min = 1.392 us, total = 1.804 s
	RayletLoadPulled - 120 total (1 active), Execution time: mean = 265.091 us, total = 31.811 ms, Queueing time: mean = 225.408 us, max = 16.997 ms, min = 37.433 us, total = 27.049 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 117 total (0 active), Execution time: mean = 83.200 us, total = 9.734 ms, Queueing time: mean = 110.887 us, max = 179.178 us, min = 26.249 us, total = 12.974 ms
	NodeManagerService.grpc_client.GetResourceLoad - 117 total (0 active), Execution time: mean = 1.542 ms, total = 180.389 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 93 total (0 active), Execution time: mean = 28.001 us, total = 2.604 ms, Queueing time: mean = 17.516 us, max = 74.560 us, min = 2.273 us, total = 1.629 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 92 total (0 active), Execution time: mean = 46.698 us, total = 4.296 ms, Queueing time: mean = 102.299 us, max = 273.037 us, min = 3.930 us, total = 9.411 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 92 total (0 active), Execution time: mean = 636.875 us, total = 58.593 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 62 total (0 active), Execution time: mean = 621.485 us, total = 38.532 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 62 total (0 active), Execution time: mean = 43.850 us, total = 2.719 ms, Queueing time: mean = 111.802 us, max = 170.409 us, min = 37.537 us, total = 6.932 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 32.295 us, total = 1.550 ms, Queueing time: mean = 126.789 us, max = 239.706 us, min = 32.077 us, total = 6.086 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 602.537 us, total = 28.922 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 40 total (1 active), Execution time: mean = 10.809 us, total = 432.348 us, Queueing time: mean = 504.061 us, max = 17.165 ms, min = 34.754 us, total = 20.162 ms
	HealthCheck - 38 total (0 active), Execution time: mean = 9.785 us, total = 371.817 us, Queueing time: mean = 101.541 us, max = 142.815 us, min = 26.125 us, total = 3.859 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 93.261 us, total = 2.238 ms, Queueing time: mean = 136.118 us, max = 183.596 us, min = 105.168 us, total = 3.267 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 742.818 us, total = 17.828 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.282 ms, total = 15.379 ms, Queueing time: mean = 76.348 us, max = 201.348 us, min = 49.832 us, total = 916.177 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsResourceManager::Update - 5 total (0 active), Execution time: mean = 101.059 us, total = 505.294 us, Queueing time: mean = 137.447 us, max = 174.838 us, min = 113.044 us, total = 687.235 us
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 841.707 us, total = 1.683 ms, Queueing time: mean = 63.856 us, max = 127.713 us, min = 127.713 us, total = 127.713 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us


[2025-01-20 22:55:01,735 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5428 total (1 active)
Queueing time: mean = 53.567 us, max = 3.876 ms, min = 3.153 us, total = 290.764 ms
Execution time:  mean = 266.149 us, total = 1.445 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 2460 total (0 active), Execution time: mean = 553.745 us, total = 1.362 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2460 total (0 active), Execution time: mean = 27.649 us, total = 68.017 ms, Queueing time: mean = 113.922 us, max = 295.599 us, min = 7.657 us, total = 280.249 ms
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 29.310 us, total = 14.128 ms, Queueing time: mean = 17.707 us, max = 3.876 ms, min = 3.153 us, total = 8.535 ms
	GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 11.467 us, total = 286.663 us, Queueing time: mean = 75.288 us, max = 140.561 us, min = 39.634 us, total = 1.882 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:55:01,735 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 722 total (25 active)
Queueing time: mean = 26.990 us, max = 531.471 us, min = 3.397 us, total = 19.487 ms
Execution time:  mean = 157.789 ms, total = 113.924 s
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 34.368 us, total = 16.565 ms, Queueing time: mean = 11.240 us, max = 531.471 us, min = 3.397 us, total = 5.418 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 59 total (0 active), Execution time: mean = 38.566 us, total = 2.275 ms, Queueing time: mean = 125.040 us, max = 279.534 us, min = 27.936 us, total = 7.377 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 59 total (24 active), Execution time: mean = 1.930 s, total = 113.862 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 35 total (0 active), Execution time: mean = 89.839 us, total = 3.144 ms, Queueing time: mean = 114.327 us, max = 237.903 us, min = 21.432 us, total = 4.001 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 35 total (0 active), Execution time: mean = 642.400 us, total = 22.484 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:55:01,735 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 494 total (0 active)
Queueing time: mean = 33.012 us, max = 10.161 ms, min = 362.000 ns, total = 16.308 ms
Execution time:  mean = 26.494 us, total = 13.088 ms
Event stats:
	event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.751 us, total = 12.412 ms, Queueing time: mean = 30.379 us, max = 10.161 ms, min = 4.223 us, total = 14.643 ms
	RaySyncer.BroadcastMessage - 5 total (0 active), Execution time: mean = 29.941 us, total = 149.703 us, Queueing time: mean = 1.013 us, max = 1.648 us, min = 362.000 ns, total = 5.063 us
	 - 5 total (0 active), Execution time: mean = 104.867 us, total = 524.334 us, Queueing time: mean = 124.732 us, max = 151.168 us, min = 105.379 us, total = 623.660 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 22:56:01,735 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 36

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 47
- cumulative published bytes: 33317
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:56:01,736 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 2587 total (4 active)
Queueing time: mean = 957.766 us, max = 256.944 ms, min = 761.000 ns, total = 2.478 s
Execution time:  mean = 357.110 us, total = 923.843 ms
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.648 us, total = 18.518 ms, Queueing time: mean = 30.797 us, max = 14.831 ms, min = 4.053 us, total = 22.235 ms
	GcsInMemoryStore.Put - 334 total (0 active), Execution time: mean = 778.303 us, total = 259.953 ms, Queueing time: mean = 5.416 ms, max = 256.536 ms, min = 1.392 us, total = 1.809 s
	RayletLoadPulled - 180 total (1 active), Execution time: mean = 254.354 us, total = 45.784 ms, Queueing time: mean = 176.851 us, max = 16.997 ms, min = 27.561 us, total = 31.833 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 177 total (0 active), Execution time: mean = 88.035 us, total = 15.582 ms, Queueing time: mean = 110.590 us, max = 208.955 us, min = 26.249 us, total = 19.574 ms
	NodeManagerService.grpc_client.GetResourceLoad - 177 total (0 active), Execution time: mean = 1.520 ms, total = 269.046 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 105 total (0 active), Execution time: mean = 27.476 us, total = 2.885 ms, Queueing time: mean = 17.218 us, max = 74.560 us, min = 2.273 us, total = 1.808 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 104 total (0 active), Execution time: mean = 47.186 us, total = 4.907 ms, Queueing time: mean = 104.738 us, max = 273.037 us, min = 3.930 us, total = 10.893 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 104 total (0 active), Execution time: mean = 631.340 us, total = 65.659 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 86 total (0 active), Execution time: mean = 603.115 us, total = 51.868 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 86 total (0 active), Execution time: mean = 44.029 us, total = 3.787 ms, Queueing time: mean = 111.050 us, max = 183.057 us, min = 37.537 us, total = 9.550 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 31.874 us, total = 2.295 ms, Queueing time: mean = 126.829 us, max = 239.706 us, min = 32.077 us, total = 9.132 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 606.130 us, total = 43.641 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 60 total (1 active), Execution time: mean = 10.643 us, total = 638.593 us, Queueing time: mean = 361.985 us, max = 17.165 ms, min = 34.754 us, total = 21.719 ms
	HealthCheck - 58 total (0 active), Execution time: mean = 9.248 us, total = 536.359 us, Queueing time: mean = 103.747 us, max = 183.109 us, min = 26.125 us, total = 6.017 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 91.435 us, total = 3.292 ms, Queueing time: mean = 126.189 us, max = 183.596 us, min = 37.894 us, total = 4.543 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 729.511 us, total = 26.262 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.346 ms, total = 24.237 ms, Queueing time: mean = 74.602 us, max = 201.348 us, min = 40.004 us, total = 1.343 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 948.924 us, total = 2.847 ms, Queueing time: mean = 61.126 us, max = 127.713 us, min = 55.666 us, total = 183.379 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us


[2025-01-20 22:56:01,736 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 8196 total (1 active)
Queueing time: mean = 53.134 us, max = 3.876 ms, min = 3.033 us, total = 435.486 ms
Execution time:  mean = 267.598 us, total = 2.193 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 3718 total (0 active), Execution time: mean = 556.271 us, total = 2.068 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3718 total (0 active), Execution time: mean = 27.918 us, total = 103.801 ms, Queueing time: mean = 113.397 us, max = 302.747 us, min = 7.657 us, total = 421.611 ms
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.758 us, total = 20.764 ms, Queueing time: mean = 15.022 us, max = 3.876 ms, min = 3.033 us, total = 10.846 ms
	GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 11.948 us, total = 442.071 us, Queueing time: mean = 79.205 us, max = 140.561 us, min = 39.634 us, total = 2.931 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:56:01,737 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1010 total (25 active)
Queueing time: mean = 24.412 us, max = 531.471 us, min = 3.397 us, total = 24.656 ms
Execution time:  mean = 172.307 ms, total = 174.030 s
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 32.771 us, total = 23.661 ms, Queueing time: mean = 11.023 us, max = 531.471 us, min = 3.397 us, total = 7.958 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 71 total (0 active), Execution time: mean = 39.852 us, total = 2.829 ms, Queueing time: mean = 122.862 us, max = 279.534 us, min = 27.936 us, total = 8.723 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 71 total (24 active), Execution time: mean = 2.450 s, total = 173.951 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 47 total (0 active), Execution time: mean = 92.810 us, total = 4.362 ms, Queueing time: mean = 112.437 us, max = 237.903 us, min = 21.432 us, total = 5.285 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 47 total (0 active), Execution time: mean = 651.012 us, total = 30.598 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:56:01,737 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 744 total (0 active)
Queueing time: mean = 27.033 us, max = 10.161 ms, min = 362.000 ns, total = 20.113 ms
Execution time:  mean = 29.854 us, total = 22.211 ms
Event stats:
	event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.463 us, total = 20.551 ms, Queueing time: mean = 24.550 us, max = 10.161 ms, min = 4.223 us, total = 17.725 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 22:57:01,737 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 48

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 59
- cumulative published bytes: 43133
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:57:01,738 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 3306 total (4 active)
Queueing time: mean = 759.544 us, max = 256.944 ms, min = 761.000 ns, total = 2.511 s
Execution time:  mean = 332.384 us, total = 1.099 s
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.087 us, total = 24.133 ms, Queueing time: mean = 25.319 us, max = 14.831 ms, min = 3.330 us, total = 24.357 ms
	GcsInMemoryStore.Put - 430 total (0 active), Execution time: mean = 605.902 us, total = 260.538 ms, Queueing time: mean = 4.224 ms, max = 256.536 ms, min = 1.392 us, total = 1.816 s
	RayletLoadPulled - 240 total (1 active), Execution time: mean = 252.647 us, total = 60.635 ms, Queueing time: mean = 152.763 us, max = 16.997 ms, min = 27.561 us, total = 36.663 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 237 total (0 active), Execution time: mean = 85.551 us, total = 20.275 ms, Queueing time: mean = 109.941 us, max = 208.955 us, min = 26.249 us, total = 26.056 ms
	NodeManagerService.grpc_client.GetResourceLoad - 237 total (0 active), Execution time: mean = 1.524 ms, total = 361.118 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 117 total (0 active), Execution time: mean = 27.039 us, total = 3.164 ms, Queueing time: mean = 16.737 us, max = 74.560 us, min = 2.273 us, total = 1.958 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 116 total (0 active), Execution time: mean = 47.908 us, total = 5.557 ms, Queueing time: mean = 104.709 us, max = 273.037 us, min = 3.930 us, total = 12.146 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 116 total (0 active), Execution time: mean = 624.322 us, total = 72.421 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 110 total (0 active), Execution time: mean = 592.896 us, total = 65.219 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 110 total (0 active), Execution time: mean = 43.059 us, total = 4.736 ms, Queueing time: mean = 108.005 us, max = 183.057 us, min = 26.281 us, total = 11.881 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 602.942 us, total = 57.882 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 32.648 us, total = 3.134 ms, Queueing time: mean = 124.999 us, max = 239.706 us, min = 32.077 us, total = 12.000 ms
	ClusterResourceManager.ResetRemoteNodeView - 80 total (1 active), Execution time: mean = 10.745 us, total = 859.628 us, Queueing time: mean = 290.626 us, max = 17.165 ms, min = 23.737 us, total = 23.250 ms
	HealthCheck - 78 total (0 active), Execution time: mean = 9.061 us, total = 706.787 us, Queueing time: mean = 106.621 us, max = 210.308 us, min = 26.125 us, total = 8.316 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 89.968 us, total = 4.318 ms, Queueing time: mean = 124.303 us, max = 183.596 us, min = 37.894 us, total = 5.967 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 715.668 us, total = 34.352 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.401 ms, total = 33.622 ms, Queueing time: mean = 75.943 us, max = 201.348 us, min = 14.980 us, total = 1.823 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.021 ms, total = 4.085 ms, Queueing time: mean = 60.523 us, max = 127.713 us, min = 55.666 us, total = 242.090 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:57:01,738 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 10966 total (1 active)
Queueing time: mean = 53.095 us, max = 3.876 ms, min = 3.033 us, total = 582.244 ms
Execution time:  mean = 267.300 us, total = 2.931 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 4977 total (0 active), Execution time: mean = 555.221 us, total = 2.763 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4977 total (0 active), Execution time: mean = 28.047 us, total = 139.590 ms, Queueing time: mean = 113.519 us, max = 748.825 us, min = 7.657 us, total = 564.985 ms
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 28.781 us, total = 27.687 ms, Queueing time: mean = 13.704 us, max = 3.876 ms, min = 3.033 us, total = 13.183 ms
	GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 12.071 us, total = 591.484 us, Queueing time: mean = 81.162 us, max = 144.269 us, min = 36.813 us, total = 3.977 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:57:01,738 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1298 total (25 active)
Queueing time: mean = 22.888 us, max = 531.471 us, min = 3.397 us, total = 29.708 ms
Execution time:  mean = 180.449 ms, total = 234.223 s
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 31.938 us, total = 30.724 ms, Queueing time: mean = 10.896 us, max = 531.471 us, min = 3.397 us, total = 10.482 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 83 total (0 active), Execution time: mean = 40.133 us, total = 3.331 ms, Queueing time: mean = 120.042 us, max = 279.534 us, min = 21.452 us, total = 9.964 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 83 total (24 active), Execution time: mean = 2.821 s, total = 234.127 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 59 total (0 active), Execution time: mean = 96.831 us, total = 5.713 ms, Queueing time: mean = 111.403 us, max = 237.903 us, min = 21.432 us, total = 6.573 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 59 total (0 active), Execution time: mean = 654.940 us, total = 38.641 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:57:01,738 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 984 total (0 active)
Queueing time: mean = 23.445 us, max = 10.161 ms, min = 362.000 ns, total = 23.070 ms
Execution time:  mean = 30.834 us, total = 30.341 ms
Event stats:
	event_loop_lag_probe - 962 total (0 active), Execution time: mean = 29.813 us, total = 28.680 ms, Queueing time: mean = 21.500 us, max = 10.161 ms, min = 4.223 us, total = 20.683 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 22:58:01,739 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 60

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 71
- cumulative published bytes: 52949
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:58:01,740 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 4025 total (4 active)
Queueing time: mean = 631.519 us, max = 256.944 ms, min = 761.000 ns, total = 2.542 s
Execution time:  mean = 316.485 us, total = 1.274 s
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 24.912 us, total = 29.944 ms, Queueing time: mean = 22.162 us, max = 14.831 ms, min = 3.330 us, total = 26.638 ms
	GcsInMemoryStore.Put - 526 total (0 active), Execution time: mean = 496.535 us, total = 261.177 ms, Queueing time: mean = 3.462 ms, max = 256.536 ms, min = 1.392 us, total = 1.821 s
	RayletLoadPulled - 300 total (1 active), Execution time: mean = 253.518 us, total = 76.055 ms, Queueing time: mean = 139.102 us, max = 16.997 ms, min = 27.561 us, total = 41.731 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 297 total (0 active), Execution time: mean = 84.018 us, total = 24.953 ms, Queueing time: mean = 109.106 us, max = 208.955 us, min = 26.249 us, total = 32.404 ms
	NodeManagerService.grpc_client.GetResourceLoad - 297 total (0 active), Execution time: mean = 1.520 ms, total = 451.552 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 134 total (0 active), Execution time: mean = 585.592 us, total = 78.469 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 134 total (0 active), Execution time: mean = 42.516 us, total = 5.697 ms, Queueing time: mean = 105.992 us, max = 183.057 us, min = 26.281 us, total = 14.203 ms
	GcsInMemoryStore.Get - 129 total (0 active), Execution time: mean = 27.150 us, total = 3.502 ms, Queueing time: mean = 16.415 us, max = 74.560 us, min = 2.273 us, total = 2.118 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 128 total (0 active), Execution time: mean = 48.366 us, total = 6.191 ms, Queueing time: mean = 104.495 us, max = 273.037 us, min = 3.930 us, total = 13.375 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 128 total (0 active), Execution time: mean = 622.078 us, total = 79.626 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 33.206 us, total = 3.985 ms, Queueing time: mean = 123.225 us, max = 239.706 us, min = 32.077 us, total = 14.787 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 601.894 us, total = 72.227 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 100 total (1 active), Execution time: mean = 11.300 us, total = 1.130 ms, Queueing time: mean = 250.636 us, max = 17.165 ms, min = 23.737 us, total = 25.064 ms
	HealthCheck - 98 total (0 active), Execution time: mean = 9.074 us, total = 889.298 us, Queueing time: mean = 107.264 us, max = 220.618 us, min = 26.125 us, total = 10.512 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 708.014 us, total = 42.481 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 91.337 us, total = 5.480 ms, Queueing time: mean = 124.930 us, max = 185.638 us, min = 37.894 us, total = 7.496 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.420 ms, total = 42.599 ms, Queueing time: mean = 78.382 us, max = 201.348 us, min = 14.980 us, total = 2.351 ms
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.158 ms, total = 5.788 ms, Queueing time: mean = 62.723 us, max = 127.713 us, min = 55.666 us, total = 313.614 us
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:58:01,740 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 13738 total (1 active)
Queueing time: mean = 53.065 us, max = 3.876 ms, min = 3.033 us, total = 729.003 ms
Execution time:  mean = 267.190 us, total = 3.671 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 6237 total (0 active), Execution time: mean = 554.836 us, total = 3.461 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6237 total (0 active), Execution time: mean = 27.916 us, total = 174.111 ms, Queueing time: mean = 113.535 us, max = 748.825 us, min = 7.173 us, total = 708.115 ms
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 29.362 us, total = 35.293 ms, Queueing time: mean = 13.188 us, max = 3.876 ms, min = 3.033 us, total = 15.852 ms
	GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 11.985 us, total = 731.080 us, Queueing time: mean = 80.953 us, max = 144.269 us, min = 36.813 us, total = 4.938 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:58:01,740 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1587 total (25 active)
Queueing time: mean = 22.025 us, max = 531.471 us, min = 3.397 us, total = 34.954 ms
Execution time:  mean = 185.540 ms, total = 294.451 s
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 31.437 us, total = 37.787 ms, Queueing time: mean = 10.849 us, max = 531.471 us, min = 3.397 us, total = 13.040 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 95 total (0 active), Execution time: mean = 42.306 us, total = 4.019 ms, Queueing time: mean = 116.809 us, max = 279.534 us, min = 21.452 us, total = 11.097 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 95 total (24 active), Execution time: mean = 3.098 s, total = 294.338 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 71 total (0 active), Execution time: mean = 103.899 us, total = 7.377 ms, Queueing time: mean = 112.712 us, max = 237.903 us, min = 21.432 us, total = 8.003 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 71 total (0 active), Execution time: mean = 665.039 us, total = 47.218 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 10.807 us, total = 21.614 us, Queueing time: mean = 62.127 us, max = 124.255 us, min = 124.255 us, total = 124.255 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us


[2025-01-20 22:58:01,740 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1224 total (0 active)
Queueing time: mean = 20.854 us, max = 10.161 ms, min = 362.000 ns, total = 25.525 ms
Execution time:  mean = 30.196 us, total = 36.959 ms
Event stats:
	event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 29.367 us, total = 35.299 ms, Queueing time: mean = 19.249 us, max = 10.161 ms, min = 4.223 us, total = 23.137 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 22:59:01,740 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 72

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 83
- cumulative published bytes: 62765
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 22:59:01,741 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 4744 total (4 active)
Queueing time: mean = 542.470 us, max = 256.944 ms, min = 761.000 ns, total = 2.573 s
Execution time:  mean = 304.939 us, total = 1.447 s
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 24.817 us, total = 35.787 ms, Queueing time: mean = 20.064 us, max = 14.831 ms, min = 3.330 us, total = 28.932 ms
	GcsInMemoryStore.Put - 622 total (0 active), Execution time: mean = 420.860 us, total = 261.775 ms, Queueing time: mean = 2.935 ms, max = 256.536 ms, min = 1.392 us, total = 1.825 s
	RayletLoadPulled - 360 total (1 active), Execution time: mean = 252.762 us, total = 90.994 ms, Queueing time: mean = 128.946 us, max = 16.997 ms, min = 27.561 us, total = 46.420 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 357 total (0 active), Execution time: mean = 83.001 us, total = 29.631 ms, Queueing time: mean = 110.283 us, max = 215.906 us, min = 26.249 us, total = 39.371 ms
	NodeManagerService.grpc_client.GetResourceLoad - 357 total (0 active), Execution time: mean = 1.510 ms, total = 539.210 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 158 total (0 active), Execution time: mean = 585.946 us, total = 92.579 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 158 total (0 active), Execution time: mean = 42.370 us, total = 6.694 ms, Queueing time: mean = 107.211 us, max = 194.012 us, min = 26.281 us, total = 16.939 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 33.331 us, total = 4.800 ms, Queueing time: mean = 122.053 us, max = 239.706 us, min = 32.077 us, total = 17.576 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 597.907 us, total = 86.099 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 141 total (0 active), Execution time: mean = 26.862 us, total = 3.788 ms, Queueing time: mean = 16.174 us, max = 74.560 us, min = 2.273 us, total = 2.280 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 140 total (0 active), Execution time: mean = 49.005 us, total = 6.861 ms, Queueing time: mean = 105.181 us, max = 273.037 us, min = 3.930 us, total = 14.725 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 140 total (0 active), Execution time: mean = 621.107 us, total = 86.955 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 120 total (1 active), Execution time: mean = 11.446 us, total = 1.373 ms, Queueing time: mean = 222.395 us, max = 17.165 ms, min = 23.737 us, total = 26.687 ms
	HealthCheck - 118 total (0 active), Execution time: mean = 9.030 us, total = 1.066 ms, Queueing time: mean = 107.880 us, max = 220.618 us, min = 26.125 us, total = 12.730 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 711.074 us, total = 51.197 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 91.327 us, total = 6.576 ms, Queueing time: mean = 126.462 us, max = 185.638 us, min = 37.894 us, total = 9.105 ms
	GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.437 ms, total = 51.716 ms, Queueing time: mean = 82.368 us, max = 201.348 us, min = 14.980 us, total = 2.965 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.238 ms, total = 7.426 ms, Queueing time: mean = 68.215 us, max = 127.713 us, min = 55.666 us, total = 409.290 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 22:59:01,742 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 16510 total (1 active)
Queueing time: mean = 52.956 us, max = 3.876 ms, min = 3.033 us, total = 874.307 ms
Execution time:  mean = 267.054 us, total = 4.409 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 7497 total (0 active), Execution time: mean = 554.400 us, total = 4.156 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7497 total (0 active), Execution time: mean = 27.865 us, total = 208.901 ms, Queueing time: mean = 113.347 us, max = 748.825 us, min = 7.173 us, total = 849.765 ms
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 29.770 us, total = 42.928 ms, Queueing time: mean = 12.839 us, max = 3.876 ms, min = 3.033 us, total = 18.514 ms
	GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 11.946 us, total = 872.066 us, Queueing time: mean = 81.229 us, max = 174.292 us, min = 36.813 us, total = 5.930 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 22:59:01,742 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 1875 total (25 active)
Queueing time: mean = 21.326 us, max = 531.471 us, min = 3.397 us, total = 39.986 ms
Execution time:  mean = 189.122 ms, total = 354.603 s
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 30.765 us, total = 44.363 ms, Queueing time: mean = 10.682 us, max = 531.471 us, min = 3.397 us, total = 15.403 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 107 total (0 active), Execution time: mean = 42.240 us, total = 4.520 ms, Queueing time: mean = 117.109 us, max = 279.534 us, min = 21.452 us, total = 12.531 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 107 total (24 active), Execution time: mean = 3.313 s, total = 354.473 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 83 total (0 active), Execution time: mean = 104.624 us, total = 8.684 ms, Queueing time: mean = 111.302 us, max = 237.903 us, min = 21.432 us, total = 9.238 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 83 total (0 active), Execution time: mean = 668.885 us, total = 55.517 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 10.807 us, total = 21.614 us, Queueing time: mean = 62.127 us, max = 124.255 us, min = 124.255 us, total = 124.255 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us


[2025-01-20 22:59:01,742 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1464 total (0 active)
Queueing time: mean = 19.128 us, max = 10.161 ms, min = 362.000 ns, total = 28.003 ms
Execution time:  mean = 29.797 us, total = 43.623 ms
Event stats:
	event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 29.100 us, total = 41.962 ms, Queueing time: mean = 17.764 us, max = 10.161 ms, min = 4.223 us, total = 25.616 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 23:00:01,742 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 84

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 95
- cumulative published bytes: 72581
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 23:00:01,743 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 5463 total (4 active)
Queueing time: mean = 476.317 us, max = 256.944 ms, min = 761.000 ns, total = 2.602 s
Execution time:  mean = 296.668 us, total = 1.621 s
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 24.581 us, total = 41.345 ms, Queueing time: mean = 18.431 us, max = 14.831 ms, min = 1.695 us, total = 31.001 ms
	GcsInMemoryStore.Put - 718 total (0 active), Execution time: mean = 365.309 us, total = 262.292 ms, Queueing time: mean = 2.547 ms, max = 256.536 ms, min = 1.392 us, total = 1.829 s
	RayletLoadPulled - 420 total (1 active), Execution time: mean = 250.945 us, total = 105.397 ms, Queueing time: mean = 120.343 us, max = 16.997 ms, min = 22.476 us, total = 50.544 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 417 total (0 active), Execution time: mean = 81.904 us, total = 34.154 ms, Queueing time: mean = 109.953 us, max = 216.110 us, min = 19.636 us, total = 45.850 ms
	NodeManagerService.grpc_client.GetResourceLoad - 417 total (0 active), Execution time: mean = 1.510 ms, total = 629.565 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 182 total (0 active), Execution time: mean = 587.296 us, total = 106.888 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 182 total (0 active), Execution time: mean = 43.560 us, total = 7.928 ms, Queueing time: mean = 106.080 us, max = 194.012 us, min = 23.939 us, total = 19.307 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 168 total (0 active), Execution time: mean = 34.234 us, total = 5.751 ms, Queueing time: mean = 121.613 us, max = 239.706 us, min = 32.077 us, total = 20.431 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 168 total (0 active), Execution time: mean = 596.428 us, total = 100.200 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 153 total (0 active), Execution time: mean = 26.363 us, total = 4.034 ms, Queueing time: mean = 15.878 us, max = 74.560 us, min = 2.273 us, total = 2.429 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 152 total (0 active), Execution time: mean = 49.373 us, total = 7.505 ms, Queueing time: mean = 105.577 us, max = 273.037 us, min = 3.930 us, total = 16.048 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 152 total (0 active), Execution time: mean = 621.363 us, total = 94.447 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 140 total (1 active), Execution time: mean = 11.374 us, total = 1.592 ms, Queueing time: mean = 200.816 us, max = 17.165 ms, min = 20.341 us, total = 28.114 ms
	HealthCheck - 138 total (0 active), Execution time: mean = 8.870 us, total = 1.224 ms, Queueing time: mean = 108.097 us, max = 220.618 us, min = 26.125 us, total = 14.917 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 84 total (0 active), Execution time: mean = 712.626 us, total = 59.861 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 84 total (0 active), Execution time: mean = 90.241 us, total = 7.580 ms, Queueing time: mean = 124.956 us, max = 185.638 us, min = 28.632 us, total = 10.496 ms
	GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), Execution time: mean = 1.421 ms, total = 59.692 ms, Queueing time: mean = 80.076 us, max = 201.348 us, min = 14.980 us, total = 3.363 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), Execution time: mean = 1.305 ms, total = 9.137 ms, Queueing time: mean = 73.771 us, max = 127.713 us, min = 55.666 us, total = 516.397 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 23:00:01,743 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 19282 total (1 active)
Queueing time: mean = 52.999 us, max = 3.876 ms, min = 3.033 us, total = 1.022 s
Execution time:  mean = 267.973 us, total = 5.167 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 8757 total (0 active), Execution time: mean = 555.513 us, total = 4.865 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 8757 total (0 active), Execution time: mean = 28.579 us, total = 250.271 ms, Queueing time: mean = 113.477 us, max = 748.825 us, min = 7.173 us, total = 993.717 ms
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 30.397 us, total = 51.128 ms, Queueing time: mean = 12.514 us, max = 3.876 ms, min = 3.033 us, total = 21.048 ms
	GcsTaskManager.GcJobSummary - 85 total (1 active), Execution time: mean = 11.928 us, total = 1.014 ms, Queueing time: mean = 83.185 us, max = 174.292 us, min = 36.813 us, total = 7.071 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 23:00:01,743 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 2163 total (25 active)
Queueing time: mean = 20.969 us, max = 531.471 us, min = 3.397 us, total = 45.357 ms
Execution time:  mean = 191.746 ms, total = 414.746 s
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 30.153 us, total = 50.718 ms, Queueing time: mean = 10.536 us, max = 531.471 us, min = 3.397 us, total = 17.721 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 119 total (0 active), Execution time: mean = 42.078 us, total = 5.007 ms, Queueing time: mean = 118.277 us, max = 279.534 us, min = 21.452 us, total = 14.075 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 119 total (24 active), Execution time: mean = 3.484 s, total = 414.599 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 95 total (0 active), Execution time: mean = 106.224 us, total = 10.091 ms, Queueing time: mean = 113.122 us, max = 237.903 us, min = 21.432 us, total = 10.747 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 95 total (0 active), Execution time: mean = 676.541 us, total = 64.271 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 10.807 us, total = 21.614 us, Queueing time: mean = 62.127 us, max = 124.255 us, min = 124.255 us, total = 124.255 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us


[2025-01-20 23:00:01,743 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1704 total (0 active)
Queueing time: mean = 18.044 us, max = 10.161 ms, min = 362.000 ns, total = 30.747 ms
Execution time:  mean = 29.978 us, total = 51.082 ms
Event stats:
	event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 29.383 us, total = 49.421 ms, Queueing time: mean = 16.861 us, max = 10.161 ms, min = 4.223 us, total = 28.360 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 23:01:01,744 I 11638 11638] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 96

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 110
- cumulative published bytes: 82496
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 534
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 256
- current buffered bytes: 256

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 42
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 11
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 10
-Total num of driver tasks: 1

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-20 23:01:01,744 I 11638 11638] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 6182 total (4 active)
Queueing time: mean = 425.484 us, max = 256.944 ms, min = 761.000 ns, total = 2.630 s
Execution time:  mean = 289.685 us, total = 1.791 s
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 24.480 us, total = 47.050 ms, Queueing time: mean = 17.272 us, max = 14.831 ms, min = 1.695 us, total = 33.197 ms
	GcsInMemoryStore.Put - 814 total (0 active), Execution time: mean = 322.917 us, total = 262.854 ms, Queueing time: mean = 2.252 ms, max = 256.536 ms, min = 1.392 us, total = 1.833 s
	RayletLoadPulled - 480 total (1 active), Execution time: mean = 248.693 us, total = 119.373 ms, Queueing time: mean = 114.490 us, max = 16.997 ms, min = 20.421 us, total = 54.955 ms
	NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 477 total (0 active), Execution time: mean = 80.806 us, total = 38.545 ms, Queueing time: mean = 107.508 us, max = 216.110 us, min = 16.818 us, total = 51.281 ms
	NodeManagerService.grpc_client.GetResourceLoad - 477 total (0 active), Execution time: mean = 1.506 ms, total = 718.328 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut - 206 total (0 active), Execution time: mean = 584.892 us, total = 120.488 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 206 total (0 active), Execution time: mean = 43.332 us, total = 8.926 ms, Queueing time: mean = 107.448 us, max = 200.615 us, min = 23.939 us, total = 22.134 ms
	NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 192 total (0 active), Execution time: mean = 33.857 us, total = 6.500 ms, Queueing time: mean = 122.347 us, max = 239.706 us, min = 32.077 us, total = 23.491 ms
	NodeInfoGcsService.grpc_server.CheckAlive - 192 total (0 active), Execution time: mean = 595.335 us, total = 114.304 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 165 total (0 active), Execution time: mean = 26.866 us, total = 4.433 ms, Queueing time: mean = 15.740 us, max = 74.560 us, min = 2.273 us, total = 2.597 ms
	InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 164 total (0 active), Execution time: mean = 49.738 us, total = 8.157 ms, Queueing time: mean = 104.943 us, max = 273.037 us, min = 3.930 us, total = 17.211 ms
	InternalKVGcsService.grpc_server.InternalKVGet - 164 total (0 active), Execution time: mean = 618.880 us, total = 101.496 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 160 total (1 active), Execution time: mean = 10.998 us, total = 1.760 ms, Queueing time: mean = 182.972 us, max = 17.165 ms, min = 20.341 us, total = 29.276 ms
	HealthCheck - 158 total (0 active), Execution time: mean = 8.789 us, total = 1.389 ms, Queueing time: mean = 109.505 us, max = 220.618 us, min = 26.125 us, total = 17.302 ms
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 96 total (0 active), Execution time: mean = 709.621 us, total = 68.124 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 96 total (0 active), Execution time: mean = 91.961 us, total = 8.828 ms, Queueing time: mean = 122.682 us, max = 185.638 us, min = 28.632 us, total = 11.778 ms
	GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), Execution time: mean = 1.408 ms, total = 67.565 ms, Queueing time: mean = 78.279 us, max = 201.348 us, min = 14.980 us, total = 3.757 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 36.599 us, total = 1.244 ms, Queueing time: mean = 125.778 us, max = 217.971 us, min = 29.829 us, total = 4.276 ms
	NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 1.352 ms, total = 45.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 595.448 us, total = 12.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 31.094 us, total = 652.978 us, Queueing time: mean = 141.855 us, max = 626.362 us, min = 24.247 us, total = 2.979 ms
	GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.755 us, total = 1.118 ms, Queueing time: mean = 130.313 us, max = 174.838 us, min = 104.579 us, total = 1.303 ms
	GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), Execution time: mean = 1.326 ms, total = 10.610 ms, Queueing time: mean = 68.586 us, max = 127.713 us, min = 32.288 us, total = 548.685 us
	GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 14.325 us, total = 85.949 us, Queueing time: mean = 38.534 us, max = 46.574 us, min = 13.894 us, total = 231.202 us
	NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.318 ms, total = 11.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 476.847 us, total = 2.384 ms, Queueing time: mean = 325.286 us, max = 1.075 ms, min = 109.625 us, total = 1.626 ms
	PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 148.064 us, total = 592.256 us, Queueing time: mean = 129.681 ms, max = 256.944 ms, min = 2.306 ms, total = 518.726 ms
	InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 695.135 us, total = 695.135 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 727.589 us, total = 727.589 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 102.845 us, total = 102.845 us, Queueing time: mean = 159.315 us, max = 159.315 us, min = 159.315 us, total = 159.315 us
	GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.163 us, total = 15.163 us, Queueing time: mean = 761.000 ns, max = 761.000 ns, min = 761.000 ns, total = 761.000 ns
	InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 914.041 us, total = 914.041 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 44.771 us, total = 44.771 us, Queueing time: mean = 106.739 us, max = 106.739 us, min = 106.739 us, total = 106.739 us
	JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 180.458 us, total = 180.458 us, Queueing time: mean = 109.188 us, max = 109.188 us, min = 109.188 us, total = 109.188 us
	InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.592 us, total = 49.592 us, Queueing time: mean = 111.343 us, max = 111.343 us, min = 111.343 us, total = 111.343 us
	NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 206.752 us, total = 206.752 us, Queueing time: mean = 114.849 us, max = 114.849 us, min = 114.849 us, total = 114.849 us
	GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.946 us, total = 17.946 us, Queueing time: mean = 13.460 us, max = 13.460 us, min = 13.460 us, total = 13.460 us
	JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 578.614 us, total = 578.614 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 21.960 us, total = 21.960 us, Queueing time: mean = 13.626 us, max = 13.626 us, min = 13.626 us, total = 13.626 us
	JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.970 us, total = 29.970 us, Queueing time: mean = 137.291 us, max = 137.291 us, min = 137.291 us, total = 137.291 us
	JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 30.307 us, total = 30.307 us, Queueing time: mean = 109.446 us, max = 109.446 us, min = 109.446 us, total = 109.446 us
	NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.117 ms, total = 1.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.185 us, total = 483.185 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 751.231 us, total = 751.231 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-20 23:01:01,744 I 11638 11638] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 22054 total (1 active)
Queueing time: mean = 52.759 us, max = 3.876 ms, min = 3.033 us, total = 1.164 s
Execution time:  mean = 267.350 us, total = 5.896 s
Event stats:
	TaskInfoGcsService.grpc_server.AddTaskEventData - 10017 total (0 active), Execution time: mean = 554.115 us, total = 5.551 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 10017 total (0 active), Execution time: mean = 28.591 us, total = 286.401 ms, Queueing time: mean = 113.021 us, max = 748.825 us, min = 7.173 us, total = 1.132 s
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 30.186 us, total = 58.018 ms, Queueing time: mean = 12.183 us, max = 3.876 ms, min = 3.033 us, total = 23.416 ms
	GcsTaskManager.GcJobSummary - 97 total (1 active), Execution time: mean = 11.679 us, total = 1.133 ms, Queueing time: mean = 81.426 us, max = 174.292 us, min = 36.091 us, total = 7.898 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.002 us, total = 14.002 us, Queueing time: mean = 98.307 us, max = 98.307 us, min = 98.307 us, total = 98.307 us


[2025-01-20 23:01:01,745 I 11638 11638] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 2463 total (25 active)
Queueing time: mean = 20.598 us, max = 531.471 us, min = 3.397 us, total = 50.733 ms
Execution time:  mean = 192.838 ms, total = 474.961 s
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 29.830 us, total = 57.334 ms, Queueing time: mean = 10.410 us, max = 531.471 us, min = 3.397 us, total = 20.008 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 134 total (0 active), Execution time: mean = 42.266 us, total = 5.664 ms, Queueing time: mean = 116.327 us, max = 279.534 us, min = 21.452 us, total = 15.588 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 134 total (24 active), Execution time: mean = 3.543 s, total = 474.796 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 110 total (0 active), Execution time: mean = 105.227 us, total = 11.575 ms, Queueing time: mean = 112.029 us, max = 237.903 us, min = 21.432 us, total = 12.323 ms
	InternalPubSubGcsService.grpc_server.GcsPublish - 110 total (0 active), Execution time: mean = 670.814 us, total = 73.790 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 61.143 us, total = 1.529 ms, Queueing time: mean = 103.091 us, max = 224.897 us, min = 29.716 us, total = 2.577 ms
	InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 625.843 us, total = 15.646 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 10.807 us, total = 21.614 us, Queueing time: mean = 62.127 us, max = 124.255 us, min = 124.255 us, total = 124.255 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 26.248 us, total = 26.248 us, Queueing time: mean = 113.067 us, max = 113.067 us, min = 113.067 us, total = 113.067 us


[2025-01-20 23:01:01,745 I 11638 11638] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 1944 total (0 active)
Queueing time: mean = 17.348 us, max = 10.161 ms, min = 362.000 ns, total = 33.724 ms
Execution time:  mean = 30.455 us, total = 59.204 ms
Event stats:
	event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 29.939 us, total = 57.543 ms, Queueing time: mean = 16.304 us, max = 10.161 ms, min = 4.223 us, total = 31.337 ms
	RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 34.753 us, total = 347.528 us, Queueing time: mean = 1.010 us, max = 1.648 us, min = 362.000 ns, total = 10.104 us
	 - 10 total (0 active), Execution time: mean = 131.120 us, total = 1.311 ms, Queueing time: mean = 134.024 us, max = 230.056 us, min = 105.379 us, total = 1.340 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 886.000 ns, total = 1.772 us, Queueing time: mean = 518.442 us, max = 519.737 us, min = 517.146 us, total = 1.037 ms


[2025-01-20 23:01:49,001 I 11638 11638] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-20 23:01:49,052 I 11638 11638] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,052 I 11638 11638] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,052 I 11638 11638] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,052 I 11638 11638] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,256 I 11638 11666] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,256 I 11638 11666] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=13be7277f830f5a8b967d2a0092091c94c7576cfebf8a5fa66025fcf
[2025-01-20 23:01:49,265 I 11638 11638] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-20 23:01:49,268 I 11638 11638] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-20 23:01:49,430 I 11638 11638] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-20 23:01:49,430 I 11638 11638] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-20 23:01:49,500 I 11638 11638] (gcs_server) stats.h:120: Stats module has shutdown.